Sep 17 14:04:43.276661 ip-10-0-133-98 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 14:04:43.276673 ip-10-0-133-98 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 14:04:43.276681 ip-10-0-133-98 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 14:04:43.276859 ip-10-0-133-98 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 14:04:53.438012 ip-10-0-133-98 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 14:04:53.438029 ip-10-0-133-98 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 95af3210f5f145a6bcf458f1f1c19c48 --
Sep 17 14:06:35.603518 ip-10-0-133-98 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 14:06:35.970193 ip-10-0-133-98 kubenswrapper[2537]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 14:06:35.970193 ip-10-0-133-98 kubenswrapper[2537]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 14:06:35.970193 ip-10-0-133-98 kubenswrapper[2537]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 14:06:35.970193 ip-10-0-133-98 kubenswrapper[2537]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 14:06:35.970193 ip-10-0-133-98 kubenswrapper[2537]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 14:06:35.970855 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.970214    2537 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 14:06:35.973564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973546    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:35.973564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973563    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:35.973564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973566    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973569    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973572    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973576    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973579    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973587    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973589    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973592    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973595    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973597    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973600    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973602    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973605    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973608    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973610    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973613    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973615    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973618    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973620    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:35.973653 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973623    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973626    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973629    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973632    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973634    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973637    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973639    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973641    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973644    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973646    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973649    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973652    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973654    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973657    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973660    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973663    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973666    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973672    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973674    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973677    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:35.974105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973680    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973682    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973685    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973688    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973690    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973693    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973695    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973698    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973700    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973703    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973705    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973708    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973710    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973714    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973716    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973719    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973721    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973724    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973726    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:35.974642 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973729    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973731    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973736    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973740    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973742    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973745    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973748    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973750    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973753    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973756    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973758    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973761    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973764    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973767    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973771    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973774    2537 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973777    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973780    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973783    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:35.975103 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973786    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973789    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973791    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973796    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973800    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973803    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973806    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973808    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973811    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973814    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973817    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973819    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973822    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973824    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973827    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973830    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973832    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973835    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973837    2537 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.973840    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:35.975591 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974540    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974547    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974550    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974552    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974555    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974559    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974562    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974564    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974567    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974570    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974573    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974575    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974581    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974585    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974588    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974591    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974594    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974596    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974599    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:35.976072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974602    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974605    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974608    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974611    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974615    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974620    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974622    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974625    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974628    2537 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974631    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974633    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974635    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974639    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974642    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974644    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974647    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974650    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974653    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974655    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:35.976564 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974660    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974663    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974666    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974669    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974672    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974675    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974678    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974681    2537 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974683    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974686    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974688    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974691    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974693    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974697    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974699    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974702    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974704    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974707    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974709    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974712    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:35.977072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974714    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974717    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974719    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974722    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974724    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974727    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974730    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974732    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974735    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974737    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974740    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974743    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974745    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974748    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974751    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974754    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974756    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974759    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974761    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974763    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:35.977582 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974766    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974769    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974772    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974775    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974777    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974781    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974783    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974786    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974788    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974791    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974793    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974796    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974798    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974801    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974803    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974806    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974809    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974811    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974814    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974816    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:35.978069 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.974819    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975204    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975209    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975212    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975215    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975218    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975221    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975224    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975226    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975229    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975232    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975234    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975237    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975240    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975242    2537 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975245    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975248    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975251    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975254    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975256    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:35.978570 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975259    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975261    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975264    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975267    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975269    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975273    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975277    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975280    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975283    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975286    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975289    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975292    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975294    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975297    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975302    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975305    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975308    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975310    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975314    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975316    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:35.979056 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975319    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975322    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975324    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975328    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975331    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975334    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975337    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975340    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975343    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975345    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975348    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975351    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975353    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975356    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975359    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975361    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975364    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975366    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975369    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:35.979558 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975372    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975374    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975377    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975379    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975382    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975386    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975390    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975394    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975397    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975399    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975401    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975404    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975407    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975410    2537 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975413    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975416    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975418    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975421    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975423    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:35.980027 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975426    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975428    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975431    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975433    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975436    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975438    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975441    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975443    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975446    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975448    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975451    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975453    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975456    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975458    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975461    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975463    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975465    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975468    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975470    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:35.980612 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975473    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975477    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.975479    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975553    2537 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975585    2537 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975635    2537 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975642    2537 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975648    2537 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975652    2537 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975656    2537 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975661    2537 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975664    2537 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975667    2537 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975671    2537 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975675    2537 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975678    2537 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975682    2537 flags.go:64] FLAG: --cgroup-root=""
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975685    2537 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975688    2537 flags.go:64] FLAG: --client-ca-file=""
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975691    2537 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975694    2537 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975698    2537 flags.go:64] FLAG: --cluster-domain=""
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975701    2537 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975704    2537 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 14:06:35.981072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975707    2537 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975711    2537 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975714    2537 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975718    2537 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975721    2537 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975725    2537 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975728    2537 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975731    2537 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975734    2537 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975740    2537 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975743    2537 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975747    2537 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975750    2537 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975754    2537 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975757    2537 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975760    2537 flags.go:64] FLAG: --enable-server="true"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975763    2537 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975769    2537 flags.go:64] FLAG: --event-burst="100"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975772    2537 flags.go:64] FLAG: --event-qps="50"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975775    2537 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975778    2537 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975781    2537 flags.go:64] FLAG: --eviction-hard=""
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975785    2537 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975788    2537 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975791    2537 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 14:06:35.981668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.975794    2537 flags.go:64] FLAG: --eviction-soft=""
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976152    2537 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976156    2537 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976159    2537 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976162    2537 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976165    2537 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976168    2537 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976171    2537 flags.go:64] FLAG: --feature-gates=""
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976185    2537 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976188    2537 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976192    2537 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976195    2537 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976199    2537 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976203    2537 flags.go:64] FLAG: --help="false"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976206    2537 flags.go:64] FLAG: --hostname-override="ip-10-0-133-98.ec2.internal"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976209    2537 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976213    2537 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976216    2537 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976220    2537 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976223    2537 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976227    2537 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976229    2537 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976232    2537 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 14:06:35.982279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976235    2537 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976238    2537 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976241    2537 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976244    2537 flags.go:64] FLAG: --kube-reserved=""
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976247    2537 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976249    2537 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976252    2537 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976255    2537 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976258    2537 flags.go:64] FLAG: --lock-file=""
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976261    2537 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976264    2537 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976266    2537 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976272    2537 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976274    2537 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976277    2537 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976280    2537 flags.go:64] FLAG: --logging-format="text"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976283    2537 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976286    2537 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976289    2537 flags.go:64] FLAG: --manifest-url=""
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976292    2537 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976296    2537 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976299    2537 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976303    2537 flags.go:64] FLAG: --max-pods="110"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976306    2537 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976309    2537 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 14:06:35.982859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976312    2537 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976315    2537 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976318    2537 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976322    2537 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976325    2537 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976332    2537 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976335    2537 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976338    2537 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976341    2537 flags.go:64] FLAG: --pod-cidr=""
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976344    2537 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976349    2537 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976352    2537 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976356    2537 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976359    2537 flags.go:64] FLAG: --port="10250"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976362    2537 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976365    2537 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0822239b9e90cbab1"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976368    2537 flags.go:64] FLAG: --qos-reserved=""
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976371    2537 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976374    2537 flags.go:64] FLAG: --register-node="true"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976377    2537 flags.go:64] FLAG: --register-with-taints=""
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976380    2537 flags.go:64] FLAG: --registry-burst="10"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976383    2537 flags.go:64] FLAG: --registry-qps="5"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976386    2537 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976389    2537 flags.go:64] FLAG: --reserved-memory=""
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976392    2537 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 14:06:35.983511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976396    2537 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976399    2537 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976401    2537 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976404    2537 flags.go:64] FLAG: --runonce="false"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976407    2537 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976410    2537 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976414    2537 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976416    2537 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976419    2537 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976422    2537 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976426    2537 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976429    2537 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976432    2537 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976435    2537 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976438    2537 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976440    2537 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976443    2537 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976446    2537 flags.go:64] FLAG: --system-cgroups=""
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976449    2537 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976454    2537 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976457    2537 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976460    2537 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976464    2537 flags.go:64] FLAG: --tls-min-version=""
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976466    2537 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976469    2537 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 14:06:35.984122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976472    2537 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976475    2537 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976478    2537 flags.go:64] FLAG: --v="2"
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976482    2537 flags.go:64] FLAG: --version="false"
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976486    2537 flags.go:64] FLAG: --vmodule=""
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976490    2537 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976493    2537 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976614    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976618    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976621    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976624    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976626    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976629    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976631    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976635    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976638    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976641    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976643    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976646    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976649    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976651    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:35.985062 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976654    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976656    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976659    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976661    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976664    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976666    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976670    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976672    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976675    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976677    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976680    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976683    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976685    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976688    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976690    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976693    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976695    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976698    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976703    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976705    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:35.985685 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976709    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976713    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976717    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976722    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976725    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976728    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976731    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976734    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976739    2537 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976742    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976745    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976748    2537 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976751    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976754    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976757    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976759    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976762    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976765    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976767    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976770    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:35.986204 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976772    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976775    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976778    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976780    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976783    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976785    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976788    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976790    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976793    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976796    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976799    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976802    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976805    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976808    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976810    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976813    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976816    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976819    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976821    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:35.986732 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976824    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976829    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976832    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976835    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976838    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976841    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976844    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976847    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976850    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976852    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976855    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976857    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976860    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976863    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976865    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976868    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976870    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976873    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976875    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976878    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:35.987214 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976880    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976883    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976886    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976889    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976892    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.976894    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.976900    2537 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.986022    2537 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.986040    2537 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.986856    2537 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.986865    2537 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986926    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986931    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986934    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986938    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:35.987716 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986941    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986944    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986947    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986950    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986953    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986956    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986959    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986962    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986964    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986967    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986969    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986972    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986975    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986980    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986984    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986987    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986989    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986992    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986995    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:35.988089 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.986997    2537 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987000    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987003    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987006    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987009    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987012    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987015    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987018    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987021    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987023    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987028    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987031    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987034    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987037    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987040    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987043    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987046    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987048    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987051    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987053    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:35.988565 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987056    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987058    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987061    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987064    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987066    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987069    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987072    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987075    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987077    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987080    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987082    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987085    2537 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987087    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987090    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987093    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987095    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987098    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987101    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987103    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987106    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:35.989066 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987109    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987111    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987114    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987136    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987140    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987143    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987146    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987149    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987152    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987155    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987158    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987161    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987164    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987167    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987169    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987172    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987192    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987196    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987199    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987201    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:35.989636 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987204    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987206    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987209    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987211    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987214    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987217    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987221    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987224    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987227    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987229    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987232    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987235    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987237    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987240    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987244    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:35.990151 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987247    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.987253    2537 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987366    2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987371    2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987374    2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987378    2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987383    2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987386    2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987389    2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987392    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987395    2537 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987397    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987400    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987403    2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:35.990530 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987406    2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987409    2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987412    2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987414    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987417    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987420    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987422    2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987425    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987428    2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987430    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987433    2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987436    2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987439    2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987441    2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987444    2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987446    2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987449    2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987452    2537 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987455    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:35.990870 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987457    2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987460    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987462    2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987465    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987467    2537 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987470    2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987472    2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987475    2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987478    2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987480    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987483    2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987486    2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987488    2537 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987492    2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987496    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987499    2537 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987502    2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987505    2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987508    2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987511    2537 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:35.991348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987514    2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987516    2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987519    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987522    2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987525    2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987527    2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987530    2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987532    2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987535    2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987537    2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987540    2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987543    2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987545    2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987548    2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987550    2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987553    2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987555    2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987558    2537 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987560    2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987563    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:35.991839 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987566    2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987569    2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987572    2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987575    2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987578    2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987581    2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987585    2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987589    2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987591    2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987594    2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987597    2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987599    2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987602    2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987604    2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987607    2537 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987609    2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987612    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987615    2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987617    2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987620    2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:35.992330 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987622    2537 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987625    2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987627    2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987630    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987633    2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987635    2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987638    2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:35.987640    2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.987645    2537 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.987760    2537 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.991002    2537 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.992208    2537 server.go:1019] "Starting client certificate rotation"
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.992311    2537 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 14:06:35.992820 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.992343    2537 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 14:06:35.994871 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.994858    2537 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 14:06:35.996851 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:35.996830    2537 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 14:06:36.007259 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.007239    2537 log.go:25] "Validated CRI v1 runtime API"
Sep 17 14:06:36.010657 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.010642    2537 log.go:25] "Validated CRI v1 image API"
Sep 17 14:06:36.012203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.012173    2537 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 14:06:36.018534 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.018515    2537 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 85165b33-b289-4518-9f0f-10e9cdea3047:/dev/nvme0n1p4 ee000ac5-931d-46c7-b117-4d2fe87e2a79:/dev/nvme0n1p3]
Sep 17 14:06:36.018602 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.018534    2537 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 14:06:36.023579 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.023451    2537 manager.go:216] Machine: {Timestamp:2026-09-17 14:06:36.022022698 +0000 UTC m=+0.313136549 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2970561 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a6639710f7816b56a7d71d8bdd063 SystemUUID:ec2a6639-710f-7816-b56a-7d71d8bdd063 BootID:95af3210-f5f1-45a6-bcf4-58f1f1c19c48 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fb:f6:76:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fb:f6:76:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:e9:85:59:d9:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 14:06:36.023579 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.023575    2537 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 14:06:36.023688 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.023653    2537 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 14:06:36.027305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.027277    2537 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 14:06:36.027438 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.027308    2537 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-98.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 14:06:36.028151 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.028142    2537 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 14:06:36.028196 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.028154    2537 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 14:06:36.028196 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.028162    2537 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 14:06:36.028984 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.028973    2537 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 14:06:36.030296 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.030286    2537 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 14:06:36.034962 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.034948    2537 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 14:06:36.035038 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.035022    2537 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 14:06:36.035118 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.035107    2537 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 14:06:36.035147 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.035140    2537 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 14:06:36.035193 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.035154    2537 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 14:06:36.035193 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.035189    2537 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 14:06:36.035250 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.035199    2537 kubelet.go:398] "Adding apiserver pod source"
Sep 17 14:06:36.035250 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.035209    2537 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 14:06:36.035585 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.035569    2537 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 14:06:36.035654 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.035589    2537 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 14:06:36.039118 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.039092    2537 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 14:06:36.042127 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.042112    2537 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 14:06:36.042171 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.042144    2537 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 14:06:36.044839 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044827    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 14:06:36.044891 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044846    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 14:06:36.044891 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044852    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 14:06:36.044891 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044887    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 14:06:36.044973 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044897    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 14:06:36.044973 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044903    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 14:06:36.044973 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044916    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 14:06:36.044973 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044921    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 14:06:36.044973 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044928    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 14:06:36.044973 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044933    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 14:06:36.044973 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044942    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 14:06:36.044973 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.044947    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 14:06:36.046751 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.046741    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 14:06:36.046751 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.046751    2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 14:06:36.050617 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.050603    2537 server.go:1268] "Started kubelet"
Sep 17 14:06:36.050769 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.050717    2537 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 14:06:36.050821 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.050729    2537 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 14:06:36.050877 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.050826    2537 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 14:06:36.051456 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.051434    2537 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 14:06:36.051563 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.051500    2537 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 14:06:36.051563 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.051532    2537 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 14:06:36.051527 ip-10-0-133-98 systemd[1]: Started Kubernetes Kubelet.
Sep 17 14:06:36.052366 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.052345    2537 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 14:06:36.053001 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.052793    2537 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 14:06:36.055006 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.054974    2537 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 14:06:36.059889 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.058793    2537 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-98.ec2.internal.18d620d3d173fca0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-98.ec2.internal,UID:ip-10-0-133-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-98.ec2.internal,},FirstTimestamp:2026-09-17 14:06:36.050578592 +0000 UTC m=+0.341692445,LastTimestamp:2026-09-17 14:06:36.050578592 +0000 UTC m=+0.341692445,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-98.ec2.internal,}"
Sep 17 14:06:36.060501 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.060482    2537 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 14:06:36.060501 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.060500    2537 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 14:06:36.060642 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.060578    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:36.061319 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061300    2537 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 14:06:36.061319 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061301    2537 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 14:06:36.061468 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061325    2537 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 14:06:36.061468 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061381    2537 factory.go:55] Registering systemd factory
Sep 17 14:06:36.061468 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061394    2537 factory.go:223] Registration of the systemd container factory successfully
Sep 17 14:06:36.061468 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061451    2537 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 14:06:36.061468 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061462    2537 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 14:06:36.061682 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061610    2537 factory.go:153] Registering CRI-O factory
Sep 17 14:06:36.061682 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061623    2537 factory.go:223] Registration of the crio container factory successfully
Sep 17 14:06:36.061682 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061677    2537 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 14:06:36.061833 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061694    2537 factory.go:103] Registering Raw factory
Sep 17 14:06:36.061833 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.061705    2537 manager.go:1196] Started watching for new ooms in manager
Sep 17 14:06:36.062070 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.062056    2537 manager.go:349] Starting recovery of all containers
Sep 17 14:06:36.069093 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.069068    2537 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 14:06:36.070229 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.070200    2537 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 14:06:36.071517 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.071496    2537 manager.go:354] Recovery completed
Sep 17 14:06:36.078074 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.078060    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:36.080790 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.080771    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:36.080868 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.080803    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:36.080868 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.080818    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:36.081426 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.081414    2537 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 14:06:36.081426 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.081426    2537 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 14:06:36.081527 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.081415    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:36.081527 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.081440    2537 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 14:06:36.082110 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.082095    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:36.082193 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.082119    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:36.082193 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.082128    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:36.083063 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.083000    2537 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-98.ec2.internal.18d620d3d340f429  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-98.ec2.internal,UID:ip-10-0-133-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-98.ec2.internal,},FirstTimestamp:2026-09-17 14:06:36.080788521 +0000 UTC m=+0.371902372,LastTimestamp:2026-09-17 14:06:36.080788521 +0000 UTC m=+0.371902372,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-98.ec2.internal,}"
Sep 17 14:06:36.083555 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.083541    2537 policy_none.go:49] "None policy: Start"
Sep 17 14:06:36.083555 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.083558    2537 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 14:06:36.083652 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.083568    2537 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 14:06:36.085521 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.085510    2537 policy_none.go:47] "Start"
Sep 17 14:06:36.097526 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.097450    2537 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-98.ec2.internal.18d620d3d34148f5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-98.ec2.internal,UID:ip-10-0-133-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-98.ec2.internal,},FirstTimestamp:2026-09-17 14:06:36.080810229 +0000 UTC m=+0.371924084,LastTimestamp:2026-09-17 14:06:36.080810229 +0000 UTC m=+0.371924084,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-98.ec2.internal,}"
Sep 17 14:06:36.105410 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.105388    2537 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5bhp"
Sep 17 14:06:36.118825 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.118810    2537 manager.go:337] "Starting Device Plugin manager"
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.118852    2537 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.118866    2537 server.go:87] "Starting device plugin registration server"
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.119212    2537 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.119222    2537 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.119334    2537 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.119407    2537 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.119416    2537 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.120055    2537 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-98.ec2.internal.18d620d3d341773f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-98.ec2.internal,UID:ip-10-0-133-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-98.ec2.internal,},FirstTimestamp:2026-09-17 14:06:36.080822079 +0000 UTC m=+0.371935931,LastTimestamp:2026-09-17 14:06:36.080822079 +0000 UTC m=+0.371935931,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-98.ec2.internal,}"
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.120851    2537 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.120908    2537 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.121094    2537 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5bhp"
Sep 17 14:06:36.138256 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.133399    2537 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-98.ec2.internal.18d620d3d340f429\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-98.ec2.internal.18d620d3d340f429  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-98.ec2.internal,UID:ip-10-0-133-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-98.ec2.internal,},FirstTimestamp:2026-09-17 14:06:36.080788521 +0000 UTC m=+0.371902372,LastTimestamp:2026-09-17 14:06:36.08210768 +0000 UTC m=+0.373221531,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-98.ec2.internal,}"
Sep 17 14:06:36.138649 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.138380    2537 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2572 into the system container \"/system.slice\": <nil>"
Sep 17 14:06:36.144483 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.144412    2537 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-98.ec2.internal.18d620d3d34148f5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-98.ec2.internal.18d620d3d34148f5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-98.ec2.internal,UID:ip-10-0-133-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-98.ec2.internal,},FirstTimestamp:2026-09-17 14:06:36.080810229 +0000 UTC m=+0.371924084,LastTimestamp:2026-09-17 14:06:36.08212336 +0000 UTC m=+0.373237213,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-98.ec2.internal,}"
Sep 17 14:06:36.153724 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.153662    2537 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-98.ec2.internal.18d620d3d341773f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-98.ec2.internal.18d620d3d341773f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-98.ec2.internal,UID:ip-10-0-133-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-98.ec2.internal,},FirstTimestamp:2026-09-17 14:06:36.080822079 +0000 UTC m=+0.371935931,LastTimestamp:2026-09-17 14:06:36.082131493 +0000 UTC m=+0.373245345,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-98.ec2.internal,}"
Sep 17 14:06:36.165597 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.165539    2537 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-98.ec2.internal.18d620d3d5aea836  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-98.ec2.internal,UID:ip-10-0-133-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-133-98.ec2.internal,},FirstTimestamp:2026-09-17 14:06:36.12153247 +0000 UTC m=+0.412646324,LastTimestamp:2026-09-17 14:06:36.12153247 +0000 UTC m=+0.412646324,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-98.ec2.internal,}"
Sep 17 14:06:36.219741 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.219704    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:36.220817 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.220732    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:36.220817 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.220766    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:36.220817 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.220780    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:36.220817 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.220812    2537 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.229688 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.229619    2537 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-98.ec2.internal.18d620d3d340f429\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-98.ec2.internal.18d620d3d340f429  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-98.ec2.internal,UID:ip-10-0-133-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-98.ec2.internal,},FirstTimestamp:2026-09-17 14:06:36.080788521 +0000 UTC m=+0.371902372,LastTimestamp:2026-09-17 14:06:36.220749869 +0000 UTC m=+0.511863725,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-98.ec2.internal,}"
Sep 17 14:06:36.237500 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.237474    2537 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 14:06:36.238657 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.238639    2537 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 14:06:36.238731 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.238697    2537 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 14:06:36.238731 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.238722    2537 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 14:06:36.238822 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.238766    2537 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 14:06:36.244463 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.244444    2537 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-98.ec2.internal\" not found" node="ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.246501 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.246485    2537 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:36.278935 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.278918    2537 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-98.ec2.internal\" not found" node="ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.339075 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.339034    2537 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-98.ec2.internal"]
Sep 17 14:06:36.339218 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.339134    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:36.340720 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.340705    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:36.340817 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.340737    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:36.340817 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.340753    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:36.343188 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.343163    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:36.343304 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.343288    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.343355 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.343321    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:36.343915 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.343900    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:36.343981 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.343923    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:36.343981 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.343899    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:36.343981 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.343959    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:36.343981 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.343937    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:36.344108 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.343969    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:36.346600 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.346587    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.346657 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.346612    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:36.347342 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.347323    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:36.347395 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.347361    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:36.347395 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.347378    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:36.363223 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.363202    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6e3318f8bcf0aa565c80ccdb4bd6a32-config\") pod \"kube-apiserver-proxy-ip-10-0-133-98.ec2.internal\" (UID: \"b6e3318f8bcf0aa565c80ccdb4bd6a32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.363291 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.363229    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c38f12b112b118af7307cb9bf0061c3f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal\" (UID: \"c38f12b112b118af7307cb9bf0061c3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.363291 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.363246    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c38f12b112b118af7307cb9bf0061c3f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal\" (UID: \"c38f12b112b118af7307cb9bf0061c3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.368750 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.368735    2537 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-98.ec2.internal\" not found" node="ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.373144 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.373128    2537 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-98.ec2.internal\" not found" node="ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.445128 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.445108    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:36.446036 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.446019    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:36.446114 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.446049    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:36.446114 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.446064    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:36.446114 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.446090    2537 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.464164 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.464146    2537 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.464221 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.464167    2537 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-98.ec2.internal\": node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:36.464270 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.464263    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c38f12b112b118af7307cb9bf0061c3f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal\" (UID: \"c38f12b112b118af7307cb9bf0061c3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.464314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.464282    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6e3318f8bcf0aa565c80ccdb4bd6a32-config\") pod \"kube-apiserver-proxy-ip-10-0-133-98.ec2.internal\" (UID: \"b6e3318f8bcf0aa565c80ccdb4bd6a32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.464314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.464297    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c38f12b112b118af7307cb9bf0061c3f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal\" (UID: \"c38f12b112b118af7307cb9bf0061c3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.464401 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.464336    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c38f12b112b118af7307cb9bf0061c3f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal\" (UID: \"c38f12b112b118af7307cb9bf0061c3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.464401 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.464365    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c38f12b112b118af7307cb9bf0061c3f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal\" (UID: \"c38f12b112b118af7307cb9bf0061c3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.464474 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.464394    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6e3318f8bcf0aa565c80ccdb4bd6a32-config\") pod \"kube-apiserver-proxy-ip-10-0-133-98.ec2.internal\" (UID: \"b6e3318f8bcf0aa565c80ccdb4bd6a32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.493135 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.493074    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:36.593978 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.593939    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:36.672252 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.672205    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.678133 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.677843    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:36.694055 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.694032    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:36.794652 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.794572    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:36.895094 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.895067    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:36.976155 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.976128    2537 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:36.992477 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.992450    2537 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 14:06:36.992630 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.992611    2537 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 14:06:36.992685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:36.992628    2537 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 14:06:36.995601 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:36.995584    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:37.095853 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:37.095815    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:37.115118 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.115084    2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:37.116305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.116288    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:37.116373 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.116317    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:37.116373 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.116328    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:37.196532 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:37.196501    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:37.230969 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.230795    2537 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 14:01:36 +0000 UTC" deadline="2028-05-13 03:47:00.257730373 +0000 UTC"
Sep 17 14:06:37.230969 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.230963    2537 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14485h40m23.026771097s"
Sep 17 14:06:37.297443 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:37.297407    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:37.310469 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:37.310437    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e3318f8bcf0aa565c80ccdb4bd6a32.slice/crio-05c99b40034afc897976a6c331f9f266102907f2a3c3cb792eb7ffed26fe349c WatchSource:0}: Error finding container 05c99b40034afc897976a6c331f9f266102907f2a3c3cb792eb7ffed26fe349c: Status 404 returned error can't find the container with id 05c99b40034afc897976a6c331f9f266102907f2a3c3cb792eb7ffed26fe349c
Sep 17 14:06:37.316096 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.316076    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:06:37.341523 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:37.341500    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38f12b112b118af7307cb9bf0061c3f.slice/crio-9934eb28e58b26582de3f89e522637437fc41a1f986fe4453a929b98ebbe35b9 WatchSource:0}: Error finding container 9934eb28e58b26582de3f89e522637437fc41a1f986fe4453a929b98ebbe35b9: Status 404 returned error can't find the container with id 9934eb28e58b26582de3f89e522637437fc41a1f986fe4453a929b98ebbe35b9
Sep 17 14:06:37.398671 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:37.398622    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:37.498816 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:37.498782    2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-98.ec2.internal\" not found"
Sep 17 14:06:37.553780 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.553740    2537 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:37.560690 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.560667    2537 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:37.575478 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.575412    2537 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:37.589002 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.588983    2537 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 14:06:37.590161 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.590148    2537 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.ec2.internal"
Sep 17 14:06:37.611887 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:37.611869    2537 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 14:06:38.036355 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.036270    2537 apiserver.go:52] "Watching apiserver"
Sep 17 14:06:38.044710 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.044678    2537 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 14:06:38.047414 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.047379    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-8fgsv","openshift-multus/multus-additional-cni-plugins-x8kj2","openshift-network-operator/iptables-alerter-lk54q","openshift-ovn-kubernetes/ovnkube-node-6fn7m","kube-system/kube-apiserver-proxy-ip-10-0-133-98.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw","openshift-cluster-node-tuning-operator/tuned-qschq","openshift-image-registry/node-ca-xz96l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal","openshift-multus/multus-msstf","openshift-multus/network-metrics-daemon-kp9pm","openshift-network-diagnostics/network-check-target-kkfch","kube-system/global-pull-secret-syncer-8dc5c","kube-system/konnectivity-agent-h78nv"]
Sep 17 14:06:38.050412 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.050386    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.053659 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.053551    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:38.053659 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.053640    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:06:38.056207 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.056181    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:06:38.057288 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.057264    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 14:06:38.057499 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.057480    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:06:38.058920 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.058896    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.059935 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.059916    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 14:06:38.060170 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.060154    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 14:06:38.062215 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.062187    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bnkqp\""
Sep 17 14:06:38.062318 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.062288    2537 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 14:06:38.062318 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.062296    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 14:06:38.062435 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.062362    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8fgsv"
Sep 17 14:06:38.063520 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.063500    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pmzkn\""
Sep 17 14:06:38.065629 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.063788    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 14:06:38.065751 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.063884    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bb8x9\""
Sep 17 14:06:38.065751 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.063928    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 14:06:38.065873 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.063930    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 14:06:38.065873 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.063974    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 14:06:38.065873 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.064079    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.066053 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.064527    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 14:06:38.085137 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085012    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctf4c\" (UniqueName: \"kubernetes.io/projected/d2d8f52c-5035-471e-b6b0-c95e222670a8-kube-api-access-ctf4c\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:38.085349 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085167    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xz96l"
Sep 17 14:06:38.085349 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085177    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.085349 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085217    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-sysctl-d\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.085349 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085269    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-systemd\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.085349 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085301    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-kubelet\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.085349 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085330    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-cni-bin\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.085644 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085358    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.085644 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085381    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-device-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.085644 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085407    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d1d14525-b7fc-4b75-bae7-a22fbca657f3-agent-certs\") pod \"konnectivity-agent-h78nv\" (UID: \"d1d14525-b7fc-4b75-bae7-a22fbca657f3\") " pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:06:38.085644 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085427    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-etc-openvswitch\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.085644 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085447    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-run-openvswitch\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.085644 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085476    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-run-ovn-kubernetes\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.085644 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085504    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e47ece57-c1f5-425d-81f9-6f255a586ec0-ovn-node-metrics-cert\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.085644 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085538    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-sys-fs\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.085644 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085573    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdvnz\" (UniqueName: \"kubernetes.io/projected/264a1b16-1248-4461-90f8-d3e7106fed5e-kube-api-access-zdvnz\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.085644 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085598    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-host\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.085644 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085630    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-systemd-units\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085661    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-slash\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085699    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e47ece57-c1f5-425d-81f9-6f255a586ec0-env-overrides\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085731    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03-hosts-file\") pod \"node-resolver-8fgsv\" (UID: \"d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03\") " pod="openshift-dns/node-resolver-8fgsv"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085768    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03-tmp-dir\") pod \"node-resolver-8fgsv\" (UID: \"d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03\") " pod="openshift-dns/node-resolver-8fgsv"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085793    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-run\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085814    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5406affc-571d-4734-a31f-64bff45cceee-etc-tuned\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085839    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-run-systemd\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085870    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-log-socket\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085888    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-cni-netd\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085906    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e47ece57-c1f5-425d-81f9-6f255a586ec0-ovnkube-script-lib\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085933    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d1d14525-b7fc-4b75-bae7-a22fbca657f3-konnectivity-ca\") pod \"konnectivity-agent-h78nv\" (UID: \"d1d14525-b7fc-4b75-bae7-a22fbca657f3\") " pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085964    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-sys\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.085984    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-lib-modules\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086000    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-var-lib-kubelet\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086028    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086055    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e47ece57-c1f5-425d-81f9-6f255a586ec0-ovnkube-config\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.086314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086093    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b58kd\" (UniqueName: \"kubernetes.io/projected/d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03-kube-api-access-b58kd\") pod \"node-resolver-8fgsv\" (UID: \"d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03\") " pod="openshift-dns/node-resolver-8fgsv"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086114    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-sysconfig\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086152    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-sysctl-conf\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086171    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086196    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx2ph\" (UniqueName: \"kubernetes.io/projected/e47ece57-c1f5-425d-81f9-6f255a586ec0-kube-api-access-gx2ph\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086216    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-kubernetes\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086255    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-run-netns\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086739    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-var-lib-openvswitch\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086798    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-run-ovn\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086857    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-node-log\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086894    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-socket-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086928    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086952    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-registration-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.087098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.086982    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-modprobe-d\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.087839 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.087254    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjr9\" (UniqueName: \"kubernetes.io/projected/5406affc-571d-4734-a31f-64bff45cceee-kube-api-access-xcjr9\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.087839 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.087334    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 14:06:38.087839 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.087348    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5406affc-571d-4734-a31f-64bff45cceee-tmp\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.087839 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.087477    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7zh6m\""
Sep 17 14:06:38.087839 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.087529    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5tcgh\""
Sep 17 14:06:38.088069 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.088013    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 14:06:38.088264 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.088238    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 14:06:38.088341 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.088219    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 14:06:38.089459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.088974    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:38.089459 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.089049    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:06:38.092320 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.091986    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.092809 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.092784    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 14:06:38.092909 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.092789    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 14:06:38.093070 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.093051    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 14:06:38.093070 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.093065    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-px4kj\""
Sep 17 14:06:38.094407 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.094381    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.097125 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.097095    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:38.097259 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.097168    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:06:38.099763 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.099448    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lk54q"
Sep 17 14:06:38.100009 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.099988    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 14:06:38.100089 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.100025    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 14:06:38.100263 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.100246    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 14:06:38.100336 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.100285    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 14:06:38.100637 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.100617    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 14:06:38.101199 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.101183    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-74rlx\""
Sep 17 14:06:38.106134 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.105713    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 14:06:38.106134 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.105804    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:06:38.106134 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.105813    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ftmvf\""
Sep 17 14:06:38.106134 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.106015    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 14:06:38.106134 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.106099    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bnqtd\""
Sep 17 14:06:38.106526 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.106276    2537 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 14:06:38.106637 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.106615    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 14:06:38.151896 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.151867    2537 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rf64h"
Sep 17 14:06:38.163411 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.163384    2537 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 14:06:38.173508 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.173470    2537 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rf64h"
Sep 17 14:06:38.188424 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188381    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjr9\" (UniqueName: \"kubernetes.io/projected/5406affc-571d-4734-a31f-64bff45cceee-kube-api-access-xcjr9\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.188661 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188432    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-var-lib-kubelet\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.188661 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188463    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e47ece57-c1f5-425d-81f9-6f255a586ec0-ovn-node-metrics-cert\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.188661 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188487    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-sysctl-d\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.188661 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188527    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-systemd\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.188661 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188559    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm6w5\" (UniqueName: \"kubernetes.io/projected/f856d63c-e51d-4532-a4e1-73fc32c4c203-kube-api-access-lm6w5\") pod \"node-ca-xz96l\" (UID: \"f856d63c-e51d-4532-a4e1-73fc32c4c203\") " pod="openshift-image-registry/node-ca-xz96l"
Sep 17 14:06:38.188661 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188585    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-etc-openvswitch\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.188661 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188610    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03-tmp-dir\") pod \"node-resolver-8fgsv\" (UID: \"d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03\") " pod="openshift-dns/node-resolver-8fgsv"
Sep 17 14:06:38.188661 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188635    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b58kd\" (UniqueName: \"kubernetes.io/projected/d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03-kube-api-access-b58kd\") pod \"node-resolver-8fgsv\" (UID: \"d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03\") " pod="openshift-dns/node-resolver-8fgsv"
Sep 17 14:06:38.188661 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188662    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p45c\" (UniqueName: \"kubernetes.io/projected/409668f2-1424-47ba-954c-39a313d33e54-kube-api-access-7p45c\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188688    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f856d63c-e51d-4532-a4e1-73fc32c4c203-host\") pod \"node-ca-xz96l\" (UID: \"f856d63c-e51d-4532-a4e1-73fc32c4c203\") " pod="openshift-image-registry/node-ca-xz96l"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188712    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2b471ed4-b2c0-4b30-9d8d-dc7dbc701109-iptables-alerter-script\") pod \"iptables-alerter-lk54q\" (UID: \"2b471ed4-b2c0-4b30-9d8d-dc7dbc701109\") " pod="openshift-network-operator/iptables-alerter-lk54q"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188737    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc4rr\" (UniqueName: \"kubernetes.io/projected/2b471ed4-b2c0-4b30-9d8d-dc7dbc701109-kube-api-access-qc4rr\") pod \"iptables-alerter-lk54q\" (UID: \"2b471ed4-b2c0-4b30-9d8d-dc7dbc701109\") " pod="openshift-network-operator/iptables-alerter-lk54q"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188761    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e47ece57-c1f5-425d-81f9-6f255a586ec0-env-overrides\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188784    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/409668f2-1424-47ba-954c-39a313d33e54-system-cni-dir\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188807    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/409668f2-1424-47ba-954c-39a313d33e54-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188836    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-cnibin\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188887    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-var-lib-cni-multus\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188915    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wjfx\" (UniqueName: \"kubernetes.io/projected/f10d459f-d28f-4cd4-ba80-3cc888742346-kube-api-access-7wjfx\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188939    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-run\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188963    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-cni-netd\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.188986    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f856d63c-e51d-4532-a4e1-73fc32c4c203-serviceca\") pod \"node-ca-xz96l\" (UID: \"f856d63c-e51d-4532-a4e1-73fc32c4c203\") " pod="openshift-image-registry/node-ca-xz96l"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189010    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-run-k8s-cni-cncf-io\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189040    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16a9d26e-dfaa-4330-be84-a026c41d6f03-dbus\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189065    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189093    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-system-cni-dir\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.189194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189118    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-etc-kubernetes\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189145    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-sysctl-conf\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189173    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189212    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-kubernetes\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189235    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-run-netns\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189262    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-var-lib-openvswitch\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189287    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-run-ovn\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189640    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-node-log\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189666    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-conf-dir\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189691    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16a9d26e-dfaa-4330-be84-a026c41d6f03-kubelet-config\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189718    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-socket-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189744    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-registration-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189768    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-modprobe-d\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189794    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/409668f2-1424-47ba-954c-39a313d33e54-cni-binary-copy\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189818    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-daemon-config\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189843    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b471ed4-b2c0-4b30-9d8d-dc7dbc701109-host-slash\") pod \"iptables-alerter-lk54q\" (UID: \"2b471ed4-b2c0-4b30-9d8d-dc7dbc701109\") " pod="openshift-network-operator/iptables-alerter-lk54q"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189870    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5406affc-571d-4734-a31f-64bff45cceee-tmp\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.190042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189895    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctf4c\" (UniqueName: \"kubernetes.io/projected/d2d8f52c-5035-471e-b6b0-c95e222670a8-kube-api-access-ctf4c\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189923    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189946    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-kubelet\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189970    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-cni-bin\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.189995    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-socket-dir-parent\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190022    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190048    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-device-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190073    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d1d14525-b7fc-4b75-bae7-a22fbca657f3-agent-certs\") pod \"konnectivity-agent-h78nv\" (UID: \"d1d14525-b7fc-4b75-bae7-a22fbca657f3\") " pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190098    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-run-openvswitch\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190123    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-run-ovn-kubernetes\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190147    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03-hosts-file\") pod \"node-resolver-8fgsv\" (UID: \"d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03\") " pod="openshift-dns/node-resolver-8fgsv"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190172    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-cni-dir\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190213    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-sys-fs\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190248    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdvnz\" (UniqueName: \"kubernetes.io/projected/264a1b16-1248-4461-90f8-d3e7106fed5e-kube-api-access-zdvnz\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190272    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-host\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190298    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-systemd-units\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190326    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-slash\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.190860 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190352    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ngxf\" (UniqueName: \"kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf\") pod \"network-check-target-kkfch\" (UID: \"0dff3702-7b53-480c-ae7a-074f122eb7e8\") " pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190377    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f10d459f-d28f-4cd4-ba80-3cc888742346-cni-binary-copy\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190402    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-run-netns\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190428    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5406affc-571d-4734-a31f-64bff45cceee-etc-tuned\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190459    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-run-systemd\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190485    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-log-socket\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190510    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e47ece57-c1f5-425d-81f9-6f255a586ec0-ovnkube-script-lib\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190535    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/409668f2-1424-47ba-954c-39a313d33e54-cnibin\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190569    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d1d14525-b7fc-4b75-bae7-a22fbca657f3-konnectivity-ca\") pod \"konnectivity-agent-h78nv\" (UID: \"d1d14525-b7fc-4b75-bae7-a22fbca657f3\") " pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190598    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-sys\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190631    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-lib-modules\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190660    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-var-lib-kubelet\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190688    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e47ece57-c1f5-425d-81f9-6f255a586ec0-ovnkube-config\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190725    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/409668f2-1424-47ba-954c-39a313d33e54-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190751    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/409668f2-1424-47ba-954c-39a313d33e54-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190779    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-sysconfig\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190806    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx2ph\" (UniqueName: \"kubernetes.io/projected/e47ece57-c1f5-425d-81f9-6f255a586ec0-kube-api-access-gx2ph\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.191685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190831    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-run-multus-certs\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.192595 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190858    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:38.192595 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190887    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-var-lib-cni-bin\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.192595 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.190914    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-hostroot\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.192595 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.191988    2537 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 14:06:38.192595 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.192477    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-sysctl-d\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.192595 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.192536    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-systemd\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.192876 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.192596    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-etc-openvswitch\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.192876 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.192866    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03-tmp-dir\") pod \"node-resolver-8fgsv\" (UID: \"d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03\") " pod="openshift-dns/node-resolver-8fgsv"
Sep 17 14:06:38.193466 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.193334    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-sysctl-conf\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.193580 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.193494    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-run\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.193580 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.193543    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-cni-netd\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.193732 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.193615    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.193782 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.193735    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:38.193834 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.193806    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs podName:d2d8f52c-5035-471e-b6b0-c95e222670a8 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:38.693784351 +0000 UTC m=+2.984898208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs") pod "network-metrics-daemon-kp9pm" (UID: "d2d8f52c-5035-471e-b6b0-c95e222670a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:38.194580 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.194035    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-kubernetes\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.194580 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.194093    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-run-netns\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.194580 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.194139    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-var-lib-openvswitch\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.194580 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.194198    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-run-ovn\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.194580 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.194241    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-node-log\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.195123 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.195033    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-run-openvswitch\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.195397 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.195376    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.195465 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.195433    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-kubelet\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.195524 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.195472    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-cni-bin\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.195577 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.195525    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.195577 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.195568    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-device-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.195968 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.195946    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-sys-fs\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.196052 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.195973    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-host\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.196052 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196004    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-run-ovn-kubernetes\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.196171 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196057    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03-hosts-file\") pod \"node-resolver-8fgsv\" (UID: \"d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03\") " pod="openshift-dns/node-resolver-8fgsv"
Sep 17 14:06:38.196245 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196169    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-registration-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.196317 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196299    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/264a1b16-1248-4461-90f8-d3e7106fed5e-socket-dir\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.196371 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196328    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-lib-modules\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.196371 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196363    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-sysconfig\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.196459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196444    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-var-lib-kubelet\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.196508 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196455    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-run-systemd\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.196611 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196490    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-systemd-units\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.196611 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196498    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-log-socket\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.196706 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196617    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e47ece57-c1f5-425d-81f9-6f255a586ec0-host-slash\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.196760 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196711    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-etc-modprobe-d\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.196893 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.196764    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5406affc-571d-4734-a31f-64bff45cceee-sys\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.197027 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.197007    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e47ece57-c1f5-425d-81f9-6f255a586ec0-ovnkube-config\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.197140 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.197116    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d1d14525-b7fc-4b75-bae7-a22fbca657f3-konnectivity-ca\") pod \"konnectivity-agent-h78nv\" (UID: \"d1d14525-b7fc-4b75-bae7-a22fbca657f3\") " pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:06:38.197790 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.197659    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e47ece57-c1f5-425d-81f9-6f255a586ec0-ovnkube-script-lib\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.197911 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.197887    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e47ece57-c1f5-425d-81f9-6f255a586ec0-env-overrides\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.199663 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.199607    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e47ece57-c1f5-425d-81f9-6f255a586ec0-ovn-node-metrics-cert\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.200613 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.200591    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d1d14525-b7fc-4b75-bae7-a22fbca657f3-agent-certs\") pod \"konnectivity-agent-h78nv\" (UID: \"d1d14525-b7fc-4b75-bae7-a22fbca657f3\") " pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:06:38.202301 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.202188    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5406affc-571d-4734-a31f-64bff45cceee-etc-tuned\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.206961 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.206933    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5406affc-571d-4734-a31f-64bff45cceee-tmp\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.221133 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.221092    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdvnz\" (UniqueName: \"kubernetes.io/projected/264a1b16-1248-4461-90f8-d3e7106fed5e-kube-api-access-zdvnz\") pod \"aws-ebs-csi-driver-node-5twvw\" (UID: \"264a1b16-1248-4461-90f8-d3e7106fed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.221862 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.221810    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctf4c\" (UniqueName: \"kubernetes.io/projected/d2d8f52c-5035-471e-b6b0-c95e222670a8-kube-api-access-ctf4c\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:38.222960 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.222424    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b58kd\" (UniqueName: \"kubernetes.io/projected/d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03-kube-api-access-b58kd\") pod \"node-resolver-8fgsv\" (UID: \"d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03\") " pod="openshift-dns/node-resolver-8fgsv"
Sep 17 14:06:38.223227 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.223205    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjr9\" (UniqueName: \"kubernetes.io/projected/5406affc-571d-4734-a31f-64bff45cceee-kube-api-access-xcjr9\") pod \"tuned-qschq\" (UID: \"5406affc-571d-4734-a31f-64bff45cceee\") " pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.223545 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.223500    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx2ph\" (UniqueName: \"kubernetes.io/projected/e47ece57-c1f5-425d-81f9-6f255a586ec0-kube-api-access-gx2ph\") pod \"ovnkube-node-6fn7m\" (UID: \"e47ece57-c1f5-425d-81f9-6f255a586ec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.245298 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.245235    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal" event={"ID":"c38f12b112b118af7307cb9bf0061c3f","Type":"ContainerStarted","Data":"9934eb28e58b26582de3f89e522637437fc41a1f986fe4453a929b98ebbe35b9"}
Sep 17 14:06:38.247758 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.247724    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.ec2.internal" event={"ID":"b6e3318f8bcf0aa565c80ccdb4bd6a32","Type":"ContainerStarted","Data":"05c99b40034afc897976a6c331f9f266102907f2a3c3cb792eb7ffed26fe349c"}
Sep 17 14:06:38.291630 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291545    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-conf-dir\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.291630 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291589    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16a9d26e-dfaa-4330-be84-a026c41d6f03-kubelet-config\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:38.291855 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291663    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-conf-dir\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.291855 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291677    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16a9d26e-dfaa-4330-be84-a026c41d6f03-kubelet-config\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:38.291855 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291698    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/409668f2-1424-47ba-954c-39a313d33e54-cni-binary-copy\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.291855 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291742    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-daemon-config\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.291855 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291768    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b471ed4-b2c0-4b30-9d8d-dc7dbc701109-host-slash\") pod \"iptables-alerter-lk54q\" (UID: \"2b471ed4-b2c0-4b30-9d8d-dc7dbc701109\") " pod="openshift-network-operator/iptables-alerter-lk54q"
Sep 17 14:06:38.291855 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291798    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-socket-dir-parent\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.291855 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291829    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-cni-dir\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.291855 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291856    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngxf\" (UniqueName: \"kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf\") pod \"network-check-target-kkfch\" (UID: \"0dff3702-7b53-480c-ae7a-074f122eb7e8\") " pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291881    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f10d459f-d28f-4cd4-ba80-3cc888742346-cni-binary-copy\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291896    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b471ed4-b2c0-4b30-9d8d-dc7dbc701109-host-slash\") pod \"iptables-alerter-lk54q\" (UID: \"2b471ed4-b2c0-4b30-9d8d-dc7dbc701109\") " pod="openshift-network-operator/iptables-alerter-lk54q"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291905    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-run-netns\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291939    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-run-netns\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291944    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/409668f2-1424-47ba-954c-39a313d33e54-cnibin\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291975    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/409668f2-1424-47ba-954c-39a313d33e54-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292001    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/409668f2-1424-47ba-954c-39a313d33e54-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292029    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-run-multus-certs\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292034    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-cni-dir\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292053    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292076    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-var-lib-cni-bin\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292111    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-hostroot\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292136    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-var-lib-kubelet\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292199    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm6w5\" (UniqueName: \"kubernetes.io/projected/f856d63c-e51d-4532-a4e1-73fc32c4c203-kube-api-access-lm6w5\") pod \"node-ca-xz96l\" (UID: \"f856d63c-e51d-4532-a4e1-73fc32c4c203\") " pod="openshift-image-registry/node-ca-xz96l"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292230    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p45c\" (UniqueName: \"kubernetes.io/projected/409668f2-1424-47ba-954c-39a313d33e54-kube-api-access-7p45c\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292233    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/409668f2-1424-47ba-954c-39a313d33e54-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.291905    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-socket-dir-parent\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292255    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f856d63c-e51d-4532-a4e1-73fc32c4c203-host\") pod \"node-ca-xz96l\" (UID: \"f856d63c-e51d-4532-a4e1-73fc32c4c203\") " pod="openshift-image-registry/node-ca-xz96l"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292281    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2b471ed4-b2c0-4b30-9d8d-dc7dbc701109-iptables-alerter-script\") pod \"iptables-alerter-lk54q\" (UID: \"2b471ed4-b2c0-4b30-9d8d-dc7dbc701109\") " pod="openshift-network-operator/iptables-alerter-lk54q"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292289    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/409668f2-1424-47ba-954c-39a313d33e54-cnibin\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292306    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc4rr\" (UniqueName: \"kubernetes.io/projected/2b471ed4-b2c0-4b30-9d8d-dc7dbc701109-kube-api-access-qc4rr\") pod \"iptables-alerter-lk54q\" (UID: \"2b471ed4-b2c0-4b30-9d8d-dc7dbc701109\") " pod="openshift-network-operator/iptables-alerter-lk54q"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292324    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/409668f2-1424-47ba-954c-39a313d33e54-cni-binary-copy\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292332    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/409668f2-1424-47ba-954c-39a313d33e54-system-cni-dir\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292324    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f10d459f-d28f-4cd4-ba80-3cc888742346-multus-daemon-config\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292358    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/409668f2-1424-47ba-954c-39a313d33e54-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292363    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-run-multus-certs\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292373    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-var-lib-cni-bin\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292383    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-cnibin\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292408    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-var-lib-cni-multus\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.292423    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:38.292653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292431    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wjfx\" (UniqueName: \"kubernetes.io/projected/f10d459f-d28f-4cd4-ba80-3cc888742346-kube-api-access-7wjfx\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292457    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f856d63c-e51d-4532-a4e1-73fc32c4c203-serviceca\") pod \"node-ca-xz96l\" (UID: \"f856d63c-e51d-4532-a4e1-73fc32c4c203\") " pod="openshift-image-registry/node-ca-xz96l"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.292481    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret podName:16a9d26e-dfaa-4330-be84-a026c41d6f03 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:38.792461788 +0000 UTC m=+3.083575628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret") pod "global-pull-secret-syncer-8dc5c" (UID: "16a9d26e-dfaa-4330-be84-a026c41d6f03") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292509    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-run-k8s-cni-cncf-io\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292535    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16a9d26e-dfaa-4330-be84-a026c41d6f03-dbus\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292562    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-system-cni-dir\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292585    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-etc-kubernetes\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292613    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f10d459f-d28f-4cd4-ba80-3cc888742346-cni-binary-copy\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292656    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-etc-kubernetes\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292787    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-run-k8s-cni-cncf-io\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292789    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/409668f2-1424-47ba-954c-39a313d33e54-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292851    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-var-lib-kubelet\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292855    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16a9d26e-dfaa-4330-be84-a026c41d6f03-dbus\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292868    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f856d63c-e51d-4532-a4e1-73fc32c4c203-serviceca\") pod \"node-ca-xz96l\" (UID: \"f856d63c-e51d-4532-a4e1-73fc32c4c203\") " pod="openshift-image-registry/node-ca-xz96l"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292885    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2b471ed4-b2c0-4b30-9d8d-dc7dbc701109-iptables-alerter-script\") pod \"iptables-alerter-lk54q\" (UID: \"2b471ed4-b2c0-4b30-9d8d-dc7dbc701109\") " pod="openshift-network-operator/iptables-alerter-lk54q"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292891    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-cnibin\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292891    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/409668f2-1424-47ba-954c-39a313d33e54-system-cni-dir\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.293305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292334    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-hostroot\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.294084 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292936    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f856d63c-e51d-4532-a4e1-73fc32c4c203-host\") pod \"node-ca-xz96l\" (UID: \"f856d63c-e51d-4532-a4e1-73fc32c4c203\") " pod="openshift-image-registry/node-ca-xz96l"
Sep 17 14:06:38.294084 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.292981    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-host-var-lib-cni-multus\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.294084 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.293025    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f10d459f-d28f-4cd4-ba80-3cc888742346-system-cni-dir\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.294084 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.293319    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/409668f2-1424-47ba-954c-39a313d33e54-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.313778 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.313742    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:06:38.313778 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.313776    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:06:38.314004 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.313792    2537 projected.go:196] Error preparing data for projected volume kube-api-access-5ngxf for pod openshift-network-diagnostics/network-check-target-kkfch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:38.314004 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.313863    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf podName:0dff3702-7b53-480c-ae7a-074f122eb7e8 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:38.813841069 +0000 UTC m=+3.104954910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5ngxf" (UniqueName: "kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf") pod "network-check-target-kkfch" (UID: "0dff3702-7b53-480c-ae7a-074f122eb7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:38.316651 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.315535    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wjfx\" (UniqueName: \"kubernetes.io/projected/f10d459f-d28f-4cd4-ba80-3cc888742346-kube-api-access-7wjfx\") pod \"multus-msstf\" (UID: \"f10d459f-d28f-4cd4-ba80-3cc888742346\") " pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.316651 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.315870    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc4rr\" (UniqueName: \"kubernetes.io/projected/2b471ed4-b2c0-4b30-9d8d-dc7dbc701109-kube-api-access-qc4rr\") pod \"iptables-alerter-lk54q\" (UID: \"2b471ed4-b2c0-4b30-9d8d-dc7dbc701109\") " pod="openshift-network-operator/iptables-alerter-lk54q"
Sep 17 14:06:38.317406 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.317378    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm6w5\" (UniqueName: \"kubernetes.io/projected/f856d63c-e51d-4532-a4e1-73fc32c4c203-kube-api-access-lm6w5\") pod \"node-ca-xz96l\" (UID: \"f856d63c-e51d-4532-a4e1-73fc32c4c203\") " pod="openshift-image-registry/node-ca-xz96l"
Sep 17 14:06:38.321196 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.321161    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p45c\" (UniqueName: \"kubernetes.io/projected/409668f2-1424-47ba-954c-39a313d33e54-kube-api-access-7p45c\") pod \"multus-additional-cni-plugins-x8kj2\" (UID: \"409668f2-1424-47ba-954c-39a313d33e54\") " pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.366693 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.366662    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qschq"
Sep 17 14:06:38.378029 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:38.377992    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5406affc_571d_4734_a31f_64bff45cceee.slice/crio-fe687e7039c121b76197aae2c39b278c2869119fc3504d30361a1e731abccddb WatchSource:0}: Error finding container fe687e7039c121b76197aae2c39b278c2869119fc3504d30361a1e731abccddb: Status 404 returned error can't find the container with id fe687e7039c121b76197aae2c39b278c2869119fc3504d30361a1e731abccddb
Sep 17 14:06:38.392476 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.392443    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:06:38.402109 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:38.402079    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d14525_b7fc_4b75_bae7_a22fbca657f3.slice/crio-62829b4b57f456390d3f05a4621fb3d9e6edfc8f86636ba41e374a9fba3484d2 WatchSource:0}: Error finding container 62829b4b57f456390d3f05a4621fb3d9e6edfc8f86636ba41e374a9fba3484d2: Status 404 returned error can't find the container with id 62829b4b57f456390d3f05a4621fb3d9e6edfc8f86636ba41e374a9fba3484d2
Sep 17 14:06:38.403433 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.403404    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:06:38.409065 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.409040    2537 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:38.412690 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.412659    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8fgsv"
Sep 17 14:06:38.412832 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:38.412807    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47ece57_c1f5_425d_81f9_6f255a586ec0.slice/crio-c93f4ac45d9053306a899962e08e8e3c68311ae9238b86bcf2bf7182f9f0b12d WatchSource:0}: Error finding container c93f4ac45d9053306a899962e08e8e3c68311ae9238b86bcf2bf7182f9f0b12d: Status 404 returned error can't find the container with id c93f4ac45d9053306a899962e08e8e3c68311ae9238b86bcf2bf7182f9f0b12d
Sep 17 14:06:38.419781 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.419754    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw"
Sep 17 14:06:38.421492 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:38.421469    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bdc7d6_aa6b_4234_aa41_4b4fbf8d5d03.slice/crio-d5bd262425c3e784052538cfea546dce3cf76d2a02232acff90cf11d0ee92766 WatchSource:0}: Error finding container d5bd262425c3e784052538cfea546dce3cf76d2a02232acff90cf11d0ee92766: Status 404 returned error can't find the container with id d5bd262425c3e784052538cfea546dce3cf76d2a02232acff90cf11d0ee92766
Sep 17 14:06:38.428832 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.428808    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xz96l"
Sep 17 14:06:38.438317 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.438285    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x8kj2"
Sep 17 14:06:38.447371 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.447335    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-msstf"
Sep 17 14:06:38.454928 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.454901    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lk54q"
Sep 17 14:06:38.696003 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.695914    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:38.696151 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.696073    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:38.696222 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.696151    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs podName:d2d8f52c-5035-471e-b6b0-c95e222670a8 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:39.69613478 +0000 UTC m=+3.987248624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs") pod "network-metrics-daemon-kp9pm" (UID: "d2d8f52c-5035-471e-b6b0-c95e222670a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:38.796535 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.796490    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:38.796723 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.796661    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:38.796791 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.796751    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret podName:16a9d26e-dfaa-4330-be84-a026c41d6f03 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:39.796730111 +0000 UTC m=+4.087843968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret") pod "global-pull-secret-syncer-8dc5c" (UID: "16a9d26e-dfaa-4330-be84-a026c41d6f03") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:38.897865 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:38.897825    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngxf\" (UniqueName: \"kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf\") pod \"network-check-target-kkfch\" (UID: \"0dff3702-7b53-480c-ae7a-074f122eb7e8\") " pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:38.898055 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.897979    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:06:38.898055 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.898000    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:06:38.898055 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.898010    2537 projected.go:196] Error preparing data for projected volume kube-api-access-5ngxf for pod openshift-network-diagnostics/network-check-target-kkfch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:38.898233 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:38.898067    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf podName:0dff3702-7b53-480c-ae7a-074f122eb7e8 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:39.898053387 +0000 UTC m=+4.189167225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ngxf" (UniqueName: "kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf") pod "network-check-target-kkfch" (UID: "0dff3702-7b53-480c-ae7a-074f122eb7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:39.176975 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:39.176866    2537 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 14:01:38 +0000 UTC" deadline="2028-05-20 06:30:13.558283285 +0000 UTC"
Sep 17 14:06:39.176975 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:39.176920    2537 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14656h23m34.381366426s"
Sep 17 14:06:39.251026 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:39.250910    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8fgsv" event={"ID":"d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03","Type":"ContainerStarted","Data":"d5bd262425c3e784052538cfea546dce3cf76d2a02232acff90cf11d0ee92766"}
Sep 17 14:06:39.252469 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:39.252440    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m" event={"ID":"e47ece57-c1f5-425d-81f9-6f255a586ec0","Type":"ContainerStarted","Data":"c93f4ac45d9053306a899962e08e8e3c68311ae9238b86bcf2bf7182f9f0b12d"}
Sep 17 14:06:39.254211 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:39.254171    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h78nv" event={"ID":"d1d14525-b7fc-4b75-bae7-a22fbca657f3","Type":"ContainerStarted","Data":"62829b4b57f456390d3f05a4621fb3d9e6edfc8f86636ba41e374a9fba3484d2"}
Sep 17 14:06:39.256139 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:39.256114    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qschq" event={"ID":"5406affc-571d-4734-a31f-64bff45cceee","Type":"ContainerStarted","Data":"fe687e7039c121b76197aae2c39b278c2869119fc3504d30361a1e731abccddb"}
Sep 17 14:06:39.551221 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:39.551190    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10d459f_d28f_4cd4_ba80_3cc888742346.slice/crio-bb4dcb6e4d2cc2427388e38752227873220de1c5346387301c05d470ecbd9591 WatchSource:0}: Error finding container bb4dcb6e4d2cc2427388e38752227873220de1c5346387301c05d470ecbd9591: Status 404 returned error can't find the container with id bb4dcb6e4d2cc2427388e38752227873220de1c5346387301c05d470ecbd9591
Sep 17 14:06:39.555246 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:39.554240    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b471ed4_b2c0_4b30_9d8d_dc7dbc701109.slice/crio-93e5b4c68d6ee6d10daba4e27d7ad72f6b30bba624930e6dcf63b77c7b5911b8 WatchSource:0}: Error finding container 93e5b4c68d6ee6d10daba4e27d7ad72f6b30bba624930e6dcf63b77c7b5911b8: Status 404 returned error can't find the container with id 93e5b4c68d6ee6d10daba4e27d7ad72f6b30bba624930e6dcf63b77c7b5911b8
Sep 17 14:06:39.555800 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:39.555773    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264a1b16_1248_4461_90f8_d3e7106fed5e.slice/crio-e957e87cd925ffd3328a033766848b098f991fbd8dfe818f14de6df874b4d247 WatchSource:0}: Error finding container e957e87cd925ffd3328a033766848b098f991fbd8dfe818f14de6df874b4d247: Status 404 returned error can't find the container with id e957e87cd925ffd3328a033766848b098f991fbd8dfe818f14de6df874b4d247
Sep 17 14:06:39.558555 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:39.558531    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf856d63c_e51d_4532_a4e1_73fc32c4c203.slice/crio-aa95c9378901b701dec8c4e6c2052890e84e198464d79bd2907f1d5c449ce808 WatchSource:0}: Error finding container aa95c9378901b701dec8c4e6c2052890e84e198464d79bd2907f1d5c449ce808: Status 404 returned error can't find the container with id aa95c9378901b701dec8c4e6c2052890e84e198464d79bd2907f1d5c449ce808
Sep 17 14:06:39.560714 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:06:39.560692    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409668f2_1424_47ba_954c_39a313d33e54.slice/crio-ac376ae0df62ac50e4f924eab8aaf2fd030df59d7591f0da7443d2c1868f2831 WatchSource:0}: Error finding container ac376ae0df62ac50e4f924eab8aaf2fd030df59d7591f0da7443d2c1868f2831: Status 404 returned error can't find the container with id ac376ae0df62ac50e4f924eab8aaf2fd030df59d7591f0da7443d2c1868f2831
Sep 17 14:06:39.704921 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:39.704884    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:39.705084 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:39.705028    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:39.705144 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:39.705093    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs podName:d2d8f52c-5035-471e-b6b0-c95e222670a8 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:41.705075062 +0000 UTC m=+5.996188922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs") pod "network-metrics-daemon-kp9pm" (UID: "d2d8f52c-5035-471e-b6b0-c95e222670a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:39.805524 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:39.805441    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:39.805677 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:39.805589    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:39.805677 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:39.805650    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret podName:16a9d26e-dfaa-4330-be84-a026c41d6f03 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:41.805633142 +0000 UTC m=+6.096746998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret") pod "global-pull-secret-syncer-8dc5c" (UID: "16a9d26e-dfaa-4330-be84-a026c41d6f03") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:39.906268 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:39.906227    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngxf\" (UniqueName: \"kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf\") pod \"network-check-target-kkfch\" (UID: \"0dff3702-7b53-480c-ae7a-074f122eb7e8\") " pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:39.906490 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:39.906471    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:06:39.906569 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:39.906500    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:06:39.906569 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:39.906514    2537 projected.go:196] Error preparing data for projected volume kube-api-access-5ngxf for pod openshift-network-diagnostics/network-check-target-kkfch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:39.906662 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:39.906571    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf podName:0dff3702-7b53-480c-ae7a-074f122eb7e8 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:41.906554195 +0000 UTC m=+6.197668037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ngxf" (UniqueName: "kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf") pod "network-check-target-kkfch" (UID: "0dff3702-7b53-480c-ae7a-074f122eb7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:40.000322 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.000271    2537 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:40.177941 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.177844    2537 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 14:01:38 +0000 UTC" deadline="2028-06-29 23:49:35.766840136 +0000 UTC"
Sep 17 14:06:40.177941 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.177884    2537 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15633h42m55.588960893s"
Sep 17 14:06:40.242012 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.241982    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:40.242214 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:40.242119    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:06:40.242577 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.242555    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:40.242698 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:40.242679    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:06:40.242769 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.242745    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:40.242858 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:40.242814    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:06:40.268566 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.268468    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-msstf" event={"ID":"f10d459f-d28f-4cd4-ba80-3cc888742346","Type":"ContainerStarted","Data":"bb4dcb6e4d2cc2427388e38752227873220de1c5346387301c05d470ecbd9591"}
Sep 17 14:06:40.279378 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.279341    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.ec2.internal" event={"ID":"b6e3318f8bcf0aa565c80ccdb4bd6a32","Type":"ContainerStarted","Data":"5feaf495e9ab42e8de628a9e5e04179db448ace29bf8f0aaa4867fb113542889"}
Sep 17 14:06:40.282536 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.282503    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lk54q" event={"ID":"2b471ed4-b2c0-4b30-9d8d-dc7dbc701109","Type":"ContainerStarted","Data":"93e5b4c68d6ee6d10daba4e27d7ad72f6b30bba624930e6dcf63b77c7b5911b8"}
Sep 17 14:06:40.287469 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.287438    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xz96l" event={"ID":"f856d63c-e51d-4532-a4e1-73fc32c4c203","Type":"ContainerStarted","Data":"aa95c9378901b701dec8c4e6c2052890e84e198464d79bd2907f1d5c449ce808"}
Sep 17 14:06:40.292392 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.291773    2537 generic.go:356] "Generic (PLEG): container finished" podID="c38f12b112b118af7307cb9bf0061c3f" containerID="b6af6b61d8015590406f1bbe122df41e68b2abeb1116077a99185d8384fa6c16" exitCode=0
Sep 17 14:06:40.292392 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.291851    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal" event={"ID":"c38f12b112b118af7307cb9bf0061c3f","Type":"ContainerDied","Data":"b6af6b61d8015590406f1bbe122df41e68b2abeb1116077a99185d8384fa6c16"}
Sep 17 14:06:40.299500 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.299473    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw" event={"ID":"264a1b16-1248-4461-90f8-d3e7106fed5e","Type":"ContainerStarted","Data":"e957e87cd925ffd3328a033766848b098f991fbd8dfe818f14de6df874b4d247"}
Sep 17 14:06:40.310298 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.310269    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8kj2" event={"ID":"409668f2-1424-47ba-954c-39a313d33e54","Type":"ContainerStarted","Data":"ac376ae0df62ac50e4f924eab8aaf2fd030df59d7591f0da7443d2c1868f2831"}
Sep 17 14:06:40.320459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:40.319974    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.ec2.internal" podStartSLOduration=3.319955917 podStartE2EDuration="3.319955917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:40.29815258 +0000 UTC m=+4.589266441" watchObservedRunningTime="2026-09-17 14:06:40.319955917 +0000 UTC m=+4.611069780"
Sep 17 14:06:41.340409 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:41.340142    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal" event={"ID":"c38f12b112b118af7307cb9bf0061c3f","Type":"ContainerStarted","Data":"e99452424953789edc806c63fd3b036022ff6d664b70b0bf0cf8972e9bf376e9"}
Sep 17 14:06:41.721309 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:41.721224    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:41.721463 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:41.721430    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:41.721527 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:41.721493    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs podName:d2d8f52c-5035-471e-b6b0-c95e222670a8 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:45.721476129 +0000 UTC m=+10.012589971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs") pod "network-metrics-daemon-kp9pm" (UID: "d2d8f52c-5035-471e-b6b0-c95e222670a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:41.822495 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:41.821886    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:41.822495 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:41.822065    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:41.822495 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:41.822137    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret podName:16a9d26e-dfaa-4330-be84-a026c41d6f03 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:45.822119001 +0000 UTC m=+10.113232842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret") pod "global-pull-secret-syncer-8dc5c" (UID: "16a9d26e-dfaa-4330-be84-a026c41d6f03") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:41.923914 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:41.923265    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngxf\" (UniqueName: \"kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf\") pod \"network-check-target-kkfch\" (UID: \"0dff3702-7b53-480c-ae7a-074f122eb7e8\") " pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:41.923914 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:41.923450    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:06:41.923914 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:41.923469    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:06:41.923914 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:41.923482    2537 projected.go:196] Error preparing data for projected volume kube-api-access-5ngxf for pod openshift-network-diagnostics/network-check-target-kkfch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:41.923914 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:41.923541    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf podName:0dff3702-7b53-480c-ae7a-074f122eb7e8 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:45.923523622 +0000 UTC m=+10.214637465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ngxf" (UniqueName: "kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf") pod "network-check-target-kkfch" (UID: "0dff3702-7b53-480c-ae7a-074f122eb7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:42.241555 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:42.241513    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:42.241738 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:42.241657    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:06:42.242133 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:42.241525    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:42.242133 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:42.241912    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:06:42.242133 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:42.241942    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:42.242133 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:42.242014    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:06:44.240207 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:44.239781    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:44.240207 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:44.239815    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:44.240207 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:44.239783    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:44.240207 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:44.239921    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:06:44.240207 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:44.240017    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:06:44.240207 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:44.240111    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:06:45.752306 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:45.752263    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:45.752797 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:45.752464    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:45.752797 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:45.752533    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs podName:d2d8f52c-5035-471e-b6b0-c95e222670a8 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:53.752513036 +0000 UTC m=+18.043626880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs") pod "network-metrics-daemon-kp9pm" (UID: "d2d8f52c-5035-471e-b6b0-c95e222670a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:45.853588 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:45.853545    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:45.853768 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:45.853675    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:45.853768 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:45.853750    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret podName:16a9d26e-dfaa-4330-be84-a026c41d6f03 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:53.8537316 +0000 UTC m=+18.144845446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret") pod "global-pull-secret-syncer-8dc5c" (UID: "16a9d26e-dfaa-4330-be84-a026c41d6f03") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:45.954908 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:45.954867    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngxf\" (UniqueName: \"kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf\") pod \"network-check-target-kkfch\" (UID: \"0dff3702-7b53-480c-ae7a-074f122eb7e8\") " pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:45.955090 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:45.955051    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:06:45.955090 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:45.955080    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:06:45.955217 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:45.955095    2537 projected.go:196] Error preparing data for projected volume kube-api-access-5ngxf for pod openshift-network-diagnostics/network-check-target-kkfch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:45.955217 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:45.955164    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf podName:0dff3702-7b53-480c-ae7a-074f122eb7e8 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:53.955145235 +0000 UTC m=+18.246259094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ngxf" (UniqueName: "kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf") pod "network-check-target-kkfch" (UID: "0dff3702-7b53-480c-ae7a-074f122eb7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:46.240220 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:46.240123    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:46.240403 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:46.240264    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:06:46.240638 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:46.240536    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:46.240738 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:46.240641    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:06:46.240738 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:46.240675    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:46.240844 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:46.240750    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:06:48.239397 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:48.239364    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:48.239822 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:48.239364    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:48.239822 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:48.239503    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:06:48.239822 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:48.239364    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:48.239822 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:48.239606    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:06:48.239822 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:48.239717    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:06:50.239733 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:50.239698    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:50.239733 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:50.239737    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:50.240227 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:50.239699    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:50.240227 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:50.239824    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:06:50.240227 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:50.239923    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:06:50.240227 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:50.240018    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:06:52.239339 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:52.239303    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:52.239339 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:52.239342    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:52.239807 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:52.239307    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:52.239807 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:52.239435    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:06:52.239807 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:52.239513    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:06:52.239807 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:52.239587    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:06:53.809133 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:53.809084    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:53.809650 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:53.809261    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:53.809650 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:53.809328    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs podName:d2d8f52c-5035-471e-b6b0-c95e222670a8 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:09.809308934 +0000 UTC m=+34.100422777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs") pod "network-metrics-daemon-kp9pm" (UID: "d2d8f52c-5035-471e-b6b0-c95e222670a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:53.909917 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:53.909883    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:53.910078 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:53.910042    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:53.910136 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:53.910118    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret podName:16a9d26e-dfaa-4330-be84-a026c41d6f03 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:09.910096715 +0000 UTC m=+34.201210558 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret") pod "global-pull-secret-syncer-8dc5c" (UID: "16a9d26e-dfaa-4330-be84-a026c41d6f03") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:54.011283 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:54.011239    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngxf\" (UniqueName: \"kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf\") pod \"network-check-target-kkfch\" (UID: \"0dff3702-7b53-480c-ae7a-074f122eb7e8\") " pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:54.011455 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:54.011399    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:06:54.011455 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:54.011423    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:06:54.011455 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:54.011436    2537 projected.go:196] Error preparing data for projected volume kube-api-access-5ngxf for pod openshift-network-diagnostics/network-check-target-kkfch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:54.011584 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:54.011499    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf podName:0dff3702-7b53-480c-ae7a-074f122eb7e8 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:10.011482993 +0000 UTC m=+34.302596834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ngxf" (UniqueName: "kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf") pod "network-check-target-kkfch" (UID: "0dff3702-7b53-480c-ae7a-074f122eb7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:54.240025 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:54.239941    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:54.240025 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:54.239970    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:54.240275 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:54.239941    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:54.240275 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:54.240060    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:06:54.240275 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:54.240150    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:06:54.240275 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:54.240235    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:06:56.239689 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:56.239660    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:56.240151 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:56.239736    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:56.240151 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:56.239747    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:06:56.240151 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:56.239836    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:06:56.240151 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:56.239884    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:56.240151 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:56.239954    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:06:57.373513 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.373264    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8fgsv" event={"ID":"d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03","Type":"ContainerStarted","Data":"ed5d09b4ff826c46e38ef9d8bffdd5cc0d10a12e5f80c4a4797da74e2da865c2"}
Sep 17 14:06:57.375419 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.375400    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:06:57.375740 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.375721    2537 generic.go:356] "Generic (PLEG): container finished" podID="e47ece57-c1f5-425d-81f9-6f255a586ec0" containerID="fe0f8f0b07f8bbcdba05bb0ca8dd98b5fd4102cc755666524e0152ab769ad075" exitCode=1
Sep 17 14:06:57.375808 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.375788    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m" event={"ID":"e47ece57-c1f5-425d-81f9-6f255a586ec0","Type":"ContainerStarted","Data":"f367f9d03df6d9e1bd4e113f4abd462e9aba8f91503a07feaa939f3f8547d141"}
Sep 17 14:06:57.375884 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.375868    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m" event={"ID":"e47ece57-c1f5-425d-81f9-6f255a586ec0","Type":"ContainerStarted","Data":"98e43373a3d4da11b145e09932975bc6e145632fef874823aa2c676e8e3a3774"}
Sep 17 14:06:57.375937 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.375924    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m" event={"ID":"e47ece57-c1f5-425d-81f9-6f255a586ec0","Type":"ContainerDied","Data":"fe0f8f0b07f8bbcdba05bb0ca8dd98b5fd4102cc755666524e0152ab769ad075"}
Sep 17 14:06:57.375990 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.375977    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m" event={"ID":"e47ece57-c1f5-425d-81f9-6f255a586ec0","Type":"ContainerStarted","Data":"8cc450e9e1e048678db37f9dd1cf801382e596288de03d9e9b05f82c0e326256"}
Sep 17 14:06:57.377119 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.377101    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qschq" event={"ID":"5406affc-571d-4734-a31f-64bff45cceee","Type":"ContainerStarted","Data":"9c1c61cab45a14802a4ef1c5b19c46c5b3d75e1c4fca651eaf3b164f08d69e7f"}
Sep 17 14:06:57.378453 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.378434    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw" event={"ID":"264a1b16-1248-4461-90f8-d3e7106fed5e","Type":"ContainerStarted","Data":"2e54a7184a853ba3afd2d041ad1ea0fd1e1da1d52277305a62c32528910d6a11"}
Sep 17 14:06:57.379855 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.379837    2537 generic.go:356] "Generic (PLEG): container finished" podID="409668f2-1424-47ba-954c-39a313d33e54" containerID="ce5bb72dc0318b488c39cdf04f417ed3dd9e9bb39be6dfe21adf7896bfda27ec" exitCode=0
Sep 17 14:06:57.379914 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.379900    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8kj2" event={"ID":"409668f2-1424-47ba-954c-39a313d33e54","Type":"ContainerDied","Data":"ce5bb72dc0318b488c39cdf04f417ed3dd9e9bb39be6dfe21adf7896bfda27ec"}
Sep 17 14:06:57.381729 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.381707    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-msstf" event={"ID":"f10d459f-d28f-4cd4-ba80-3cc888742346","Type":"ContainerStarted","Data":"4605234f58a81653387a8f0f3fb879cbaecab854d04179271438d5cea61963cc"}
Sep 17 14:06:57.383450 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.383427    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h78nv" event={"ID":"d1d14525-b7fc-4b75-bae7-a22fbca657f3","Type":"ContainerStarted","Data":"288a41bace9fe545f5eddbe3d1c79c90b62c7cf15dafba3d66bbdc9072847de4"}
Sep 17 14:06:57.384989 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.384970    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xz96l" event={"ID":"f856d63c-e51d-4532-a4e1-73fc32c4c203","Type":"ContainerStarted","Data":"9c45dbe38f2c2449d66a8c88b43e73fddea18bf9d7d43cf5b7d836eb1052d2a9"}
Sep 17 14:06:57.418312 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.418121    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.ec2.internal" podStartSLOduration=20.418092263 podStartE2EDuration="20.418092263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:41.365415619 +0000 UTC m=+5.656529482" watchObservedRunningTime="2026-09-17 14:06:57.418092263 +0000 UTC m=+21.709206102"
Sep 17 14:06:57.458240 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.458201    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8fgsv" podStartSLOduration=3.052178236 podStartE2EDuration="21.458170616s" totalImagesPullingTime="18.40599238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:57.420189612 +0000 UTC m=+21.711303479" watchObservedRunningTime="2026-09-17 14:06:57.458170616 +0000 UTC m=+21.749284536"
Sep 17 14:06:57.458618 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.458593    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xz96l" podStartSLOduration=4.191998941 podStartE2EDuration="21.45858708s" totalImagesPullingTime="17.266588139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:57.458460634 +0000 UTC m=+21.749574494" watchObservedRunningTime="2026-09-17 14:06:57.45858708 +0000 UTC m=+21.749700963"
Sep 17 14:06:57.501802 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.501755    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qschq" podStartSLOduration=3.050731687 podStartE2EDuration="21.501739919s" totalImagesPullingTime="18.451008232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:57.501498185 +0000 UTC m=+21.792612037" watchObservedRunningTime="2026-09-17 14:06:57.501739919 +0000 UTC m=+21.792853780"
Sep 17 14:06:57.545955 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.545898    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h78nv" podStartSLOduration=3.12164759 podStartE2EDuration="21.545876106s" totalImagesPullingTime="18.424228516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:57.545548319 +0000 UTC m=+21.836662179" watchObservedRunningTime="2026-09-17 14:06:57.545876106 +0000 UTC m=+21.836989967"
Sep 17 14:06:57.759355 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:57.757464    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-msstf" podStartSLOduration=4.25554146 podStartE2EDuration="21.757444401s" totalImagesPullingTime="17.501902941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:57.756846534 +0000 UTC m=+22.047960396" watchObservedRunningTime="2026-09-17 14:06:57.757444401 +0000 UTC m=+22.048558263"
Sep 17 14:06:58.239358 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.239327    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:06:58.239358 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.239343    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:06:58.239548 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:58.239453    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:06:58.239548 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:58.239523    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:06:58.239620 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.239584    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:06:58.239692 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:06:58.239663    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:06:58.376553 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.376518    2537 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 14:06:58.390406 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.390375    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:06:58.390852 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.390821    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m" event={"ID":"e47ece57-c1f5-425d-81f9-6f255a586ec0","Type":"ContainerStarted","Data":"b3dec17e6f13501bb3b792f81a11f8c6c4a43ef966860df2e2a573d8f4e4fe1a"}
Sep 17 14:06:58.390964 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.390899    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m" event={"ID":"e47ece57-c1f5-425d-81f9-6f255a586ec0","Type":"ContainerStarted","Data":"f038ab86d03cf33f5826a5666ba01950e61afde99a4e41e294eac6120b8ff4a6"}
Sep 17 14:06:58.395004 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.394976    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw" event={"ID":"264a1b16-1248-4461-90f8-d3e7106fed5e","Type":"ContainerStarted","Data":"b103177d7b111b491b4e5f77a778f13dfad2850069ed1e7defca094008ac0ae0"}
Sep 17 14:06:58.396736 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.396710    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lk54q" event={"ID":"2b471ed4-b2c0-4b30-9d8d-dc7dbc701109","Type":"ContainerStarted","Data":"3069f78ffa457ac8351f78abb02c2585d07beb8e1caa1d1c41eb180eae465bb0"}
Sep 17 14:06:58.618113 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.618035    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:06:58.618599 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.618581    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:06:58.645980 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:58.645934    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lk54q" podStartSLOduration=4.374418921 podStartE2EDuration="21.645918556s" totalImagesPullingTime="17.271499635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:58.435973233 +0000 UTC m=+22.727087094" watchObservedRunningTime="2026-09-17 14:06:58.645918556 +0000 UTC m=+22.937032416"
Sep 17 14:06:59.133283 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:59.133075    2537 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T14:06:58.376541181Z","UUID":"90f04fe0-cd1a-4e87-bca5-0d9dfa7743df","Handler":null,"Name":"","Endpoint":""}
Sep 17 14:06:59.135017 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:59.134994    2537 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 14:06:59.135159 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:59.135035    2537 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 14:06:59.399262 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:59.399195    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:06:59.399775 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:06:59.399740    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h78nv"
Sep 17 14:07:00.239371 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:00.239335    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:00.239659 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:00.239336    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:07:00.239659 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:00.239439    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:07:00.239659 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:00.239528    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:07:00.239659 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:00.239336    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:07:00.239659 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:00.239608    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:07:00.405132 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:00.405106    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:07:00.405653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:00.405557    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m" event={"ID":"e47ece57-c1f5-425d-81f9-6f255a586ec0","Type":"ContainerStarted","Data":"e230b0fe43cc3e81983cd7f6130104dac86989137fefcec97ea2597d81f49a8f"}
Sep 17 14:07:00.408052 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:00.408000    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw" event={"ID":"264a1b16-1248-4461-90f8-d3e7106fed5e","Type":"ContainerStarted","Data":"af7ebd6e36aff5abaf0c283c06f6d60099472ac62fbb0828f2de3b49f8be99e4"}
Sep 17 14:07:00.449733 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:00.449675    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5twvw" podStartSLOduration=4.974877974 podStartE2EDuration="24.449653457s" totalImagesPullingTime="19.474775483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:00.447832369 +0000 UTC m=+24.738946228" watchObservedRunningTime="2026-09-17 14:07:00.449653457 +0000 UTC m=+24.740767319"
Sep 17 14:07:02.239772 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:02.239630    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:02.240056 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:02.239632    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:07:02.240056 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:02.239844    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:07:02.240056 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:02.239935    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:07:02.240056 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:02.239699    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:07:02.240056 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:02.240016    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:07:02.416266 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:02.416245    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:07:02.416735 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:02.416700    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m" event={"ID":"e47ece57-c1f5-425d-81f9-6f255a586ec0","Type":"ContainerStarted","Data":"ab2a9c7d8a4d5e732c80a994e8bfef248ad55332329f38cba1cb60f56fbc5756"}
Sep 17 14:07:02.416983 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:02.416963    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:07:02.417141 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:02.417122    2537 scope.go:117] "RemoveContainer" containerID="fe0f8f0b07f8bbcdba05bb0ca8dd98b5fd4102cc755666524e0152ab769ad075"
Sep 17 14:07:02.435394 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:02.435371    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:07:03.422442 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:03.422411    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:07:03.423285 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:03.422835    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m" event={"ID":"e47ece57-c1f5-425d-81f9-6f255a586ec0","Type":"ContainerStarted","Data":"87fb02ec3881ff0eb7847e9a4cc26523af0ded4f088c74ea9df0c5e93a561994"}
Sep 17 14:07:03.423285 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:03.423130    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:07:03.423285 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:03.423214    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:07:03.424939 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:03.424905    2537 generic.go:356] "Generic (PLEG): container finished" podID="409668f2-1424-47ba-954c-39a313d33e54" containerID="04deb32a588c01b24720a12cb32ed72c479bbc5aaa69bf41bfa1fd363a2b8831" exitCode=0
Sep 17 14:07:03.425043 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:03.424950    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8kj2" event={"ID":"409668f2-1424-47ba-954c-39a313d33e54","Type":"ContainerDied","Data":"04deb32a588c01b24720a12cb32ed72c479bbc5aaa69bf41bfa1fd363a2b8831"}
Sep 17 14:07:03.438930 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:03.438903    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:07:03.521430 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:03.521318    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m" podStartSLOduration=9.085668126 podStartE2EDuration="27.521298943s" totalImagesPullingTime="18.435630817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:03.489070455 +0000 UTC m=+27.780184318" watchObservedRunningTime="2026-09-17 14:07:03.521298943 +0000 UTC m=+27.812412786"
Sep 17 14:07:04.239962 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:04.239779    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:07:04.240171 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:04.239835    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:04.240171 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:04.240036    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:07:04.240171 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:04.240137    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:07:04.240171 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:04.239852    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:07:04.240357 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:04.240274    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:07:04.577422 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:04.577390    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kp9pm"]
Sep 17 14:07:04.577867 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:04.577512    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:07:04.577867 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:04.577635    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:07:04.583217 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:04.583166    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8dc5c"]
Sep 17 14:07:04.583349 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:04.583324    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:07:04.583449 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:04.583430    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:07:04.603992 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:04.603967    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kkfch"]
Sep 17 14:07:04.604098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:04.604084    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:04.604322 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:04.604297    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:07:05.431594 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:05.431562    2537 generic.go:356] "Generic (PLEG): container finished" podID="409668f2-1424-47ba-954c-39a313d33e54" containerID="58b9298ada3ba31d198d57f065b79d34df9d13925c712f60ee1c5c5290971b09" exitCode=0
Sep 17 14:07:05.431763 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:05.431652    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8kj2" event={"ID":"409668f2-1424-47ba-954c-39a313d33e54","Type":"ContainerDied","Data":"58b9298ada3ba31d198d57f065b79d34df9d13925c712f60ee1c5c5290971b09"}
Sep 17 14:07:06.240303 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:06.240194    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:07:06.240303 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:06.240246    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:07:06.240731 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:06.240339    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:06.240731 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:06.240418    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:07:06.240848 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:06.240777    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:07:06.240848 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:06.240797    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:07:07.438568 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:07.438536    2537 generic.go:356] "Generic (PLEG): container finished" podID="409668f2-1424-47ba-954c-39a313d33e54" containerID="9e56151431a94dbf739c30ed805d52b1461dc9fff9c4012f396ec0174c133256" exitCode=0
Sep 17 14:07:07.438992 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:07.438582    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8kj2" event={"ID":"409668f2-1424-47ba-954c-39a313d33e54","Type":"ContainerDied","Data":"9e56151431a94dbf739c30ed805d52b1461dc9fff9c4012f396ec0174c133256"}
Sep 17 14:07:08.239572 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:08.239488    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:08.239572 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:08.239488    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:07:08.239814 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:08.239611    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:07:08.239814 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:08.239601    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkfch" podUID="0dff3702-7b53-480c-ae7a-074f122eb7e8"
Sep 17 14:07:08.239814 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:08.239719    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dc5c" podUID="16a9d26e-dfaa-4330-be84-a026c41d6f03"
Sep 17 14:07:08.239814 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:08.239807    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:07:09.535555 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.535522    2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-98.ec2.internal" event="NodeReady"
Sep 17 14:07:09.535962 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.535649    2537 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 14:07:09.599447 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.599409    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-648b8d4db-5pgz2"]
Sep 17 14:07:09.602083 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.602063    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.607416 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.607392    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 14:07:09.608194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.608152    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 14:07:09.608592 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.608565    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2g4kt\""
Sep 17 14:07:09.609230 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.609209    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 14:07:09.657801 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.657766    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-648b8d4db-5pgz2"]
Sep 17 14:07:09.661454 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.661426    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 14:07:09.722484 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.722404    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-bound-sa-token\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.722484 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.722469    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-ca-trust-extracted\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.722670 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.722495    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.722670 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.722512    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-certificates\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.722670 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.722527    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kzn\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-kube-api-access-x4kzn\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.722670 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.722573    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-image-registry-private-configuration\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.722670 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.722603    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-trusted-ca\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.722841 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.722702    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-installation-pull-secrets\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.744613 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.744569    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zsfks"]
Sep 17 14:07:09.747704 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.747679    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m4qxw"]
Sep 17 14:07:09.747845 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.747804    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zsfks"
Sep 17 14:07:09.751843 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.751818    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 14:07:09.751992 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.751968    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jltzw\""
Sep 17 14:07:09.752261 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.752235    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 14:07:09.752374 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.752357    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.752486 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.752471    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 14:07:09.757014 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.756991    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 14:07:09.757014 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.757001    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jdh9k\""
Sep 17 14:07:09.757266 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.757250    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 14:07:09.764980 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.764955    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zsfks"]
Sep 17 14:07:09.771358 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.771337    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m4qxw"]
Sep 17 14:07:09.823642 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.823596    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-installation-pull-secrets\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.823824 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.823656    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-bound-sa-token\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.823824 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.823703    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-ca-trust-extracted\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.823824 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.823726    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.823824 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.823748    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-certificates\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.824023 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:09.823838    2537 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:07:09.824023 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:09.823860    2537 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648b8d4db-5pgz2: secret "image-registry-tls" not found
Sep 17 14:07:09.824023 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:09.823921    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls podName:1d4ece8e-8652-48fc-bd1e-77005ef9b11d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:10.323900388 +0000 UTC m=+34.615014240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls") pod "image-registry-648b8d4db-5pgz2" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d") : secret "image-registry-tls" not found
Sep 17 14:07:09.824023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.823940    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kzn\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-kube-api-access-x4kzn\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.824023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.823973    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/916ac6c2-3233-45cd-9c79-088264c87d00-cert\") pod \"ingress-canary-zsfks\" (UID: \"916ac6c2-3233-45cd-9c79-088264c87d00\") " pod="openshift-ingress-canary/ingress-canary-zsfks"
Sep 17 14:07:09.824023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.824008    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-image-registry-private-configuration\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.824353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.824034    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-trusted-ca\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.824353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.824065    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-config-volume\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.824353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.824095    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bz4\" (UniqueName: \"kubernetes.io/projected/916ac6c2-3233-45cd-9c79-088264c87d00-kube-api-access-p9bz4\") pod \"ingress-canary-zsfks\" (UID: \"916ac6c2-3233-45cd-9c79-088264c87d00\") " pod="openshift-ingress-canary/ingress-canary-zsfks"
Sep 17 14:07:09.824353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.824128    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:07:09.824353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.824147    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-tmp-dir\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.824353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.824170    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.824353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.824210    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnnc\" (UniqueName: \"kubernetes.io/projected/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-kube-api-access-2dnnc\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.824353 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:09.824320    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:07:09.824734 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:09.824411    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs podName:d2d8f52c-5035-471e-b6b0-c95e222670a8 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:41.824391648 +0000 UTC m=+66.115505505 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs") pod "network-metrics-daemon-kp9pm" (UID: "d2d8f52c-5035-471e-b6b0-c95e222670a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:07:09.824813 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.824787    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-ca-trust-extracted\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.824983 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.824962    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-certificates\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.825680 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.825657    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-trusted-ca\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.828962 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.828938    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-installation-pull-secrets\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.828962 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.828947    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-image-registry-private-configuration\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.840035 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.840010    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-bound-sa-token\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.842316 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.842288    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kzn\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-kube-api-access-x4kzn\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:09.925140 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.925098    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.925140 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.925148    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnnc\" (UniqueName: \"kubernetes.io/projected/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-kube-api-access-2dnnc\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.925398 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.925218    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:07:09.925398 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:09.925240    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:09.925398 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.925254    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/916ac6c2-3233-45cd-9c79-088264c87d00-cert\") pod \"ingress-canary-zsfks\" (UID: \"916ac6c2-3233-45cd-9c79-088264c87d00\") " pod="openshift-ingress-canary/ingress-canary-zsfks"
Sep 17 14:07:09.925398 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.925279    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-config-volume\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.925398 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:09.925307    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls podName:05376a4a-6b6c-46cf-b8aa-77f1e8995a27 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:10.425285472 +0000 UTC m=+34.716399327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls") pod "dns-default-m4qxw" (UID: "05376a4a-6b6c-46cf-b8aa-77f1e8995a27") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:09.925398 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.925344    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bz4\" (UniqueName: \"kubernetes.io/projected/916ac6c2-3233-45cd-9c79-088264c87d00-kube-api-access-p9bz4\") pod \"ingress-canary-zsfks\" (UID: \"916ac6c2-3233-45cd-9c79-088264c87d00\") " pod="openshift-ingress-canary/ingress-canary-zsfks"
Sep 17 14:07:09.925398 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:09.925363    2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:07:09.925398 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.925380    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-tmp-dir\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.925788 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:09.925422    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret podName:16a9d26e-dfaa-4330-be84-a026c41d6f03 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:41.92540326 +0000 UTC m=+66.216517108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret") pod "global-pull-secret-syncer-8dc5c" (UID: "16a9d26e-dfaa-4330-be84-a026c41d6f03") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:07:09.925788 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.925679    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-tmp-dir\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.925899 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.925855    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-config-volume\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.928130 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.928100    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/916ac6c2-3233-45cd-9c79-088264c87d00-cert\") pod \"ingress-canary-zsfks\" (UID: \"916ac6c2-3233-45cd-9c79-088264c87d00\") " pod="openshift-ingress-canary/ingress-canary-zsfks"
Sep 17 14:07:09.937329 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.937304    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnnc\" (UniqueName: \"kubernetes.io/projected/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-kube-api-access-2dnnc\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:09.940509 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.940490    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bz4\" (UniqueName: \"kubernetes.io/projected/916ac6c2-3233-45cd-9c79-088264c87d00-kube-api-access-p9bz4\") pod \"ingress-canary-zsfks\" (UID: \"916ac6c2-3233-45cd-9c79-088264c87d00\") " pod="openshift-ingress-canary/ingress-canary-zsfks"
Sep 17 14:07:09.968330 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:09.968290    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h78nv_d1d14525-b7fc-4b75-bae7-a22fbca657f3/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:10.026027 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.025942    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngxf\" (UniqueName: \"kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf\") pod \"network-check-target-kkfch\" (UID: \"0dff3702-7b53-480c-ae7a-074f122eb7e8\") " pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:10.026211 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:10.026137    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:07:10.026211 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:10.026165    2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:07:10.026211 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:10.026200    2537 projected.go:196] Error preparing data for projected volume kube-api-access-5ngxf for pod openshift-network-diagnostics/network-check-target-kkfch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:07:10.026379 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:10.026268    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf podName:0dff3702-7b53-480c-ae7a-074f122eb7e8 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:42.026245475 +0000 UTC m=+66.317359330 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ngxf" (UniqueName: "kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf") pod "network-check-target-kkfch" (UID: "0dff3702-7b53-480c-ae7a-074f122eb7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:07:10.063667 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.063622    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zsfks"
Sep 17 14:07:10.106752 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.106715    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h78nv_d1d14525-b7fc-4b75-bae7-a22fbca657f3/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:10.239653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.239614    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:07:10.239836 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.239613    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:10.239836 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.239618    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:07:10.242947 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.242924    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 14:07:10.243089 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.242991    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 14:07:10.243271 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.243245    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 14:07:10.243271 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.243271    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 14:07:10.243585 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.243569    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c2694\""
Sep 17 14:07:10.243673 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.243607    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ck6nq\""
Sep 17 14:07:10.328232 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.328133    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:10.328369 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:10.328314    2537 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:07:10.328369 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:10.328332    2537 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648b8d4db-5pgz2: secret "image-registry-tls" not found
Sep 17 14:07:10.328443 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:10.328401    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls podName:1d4ece8e-8652-48fc-bd1e-77005ef9b11d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:11.328386141 +0000 UTC m=+35.619499980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls") pod "image-registry-648b8d4db-5pgz2" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d") : secret "image-registry-tls" not found
Sep 17 14:07:10.429191 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:10.429145    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:10.429381 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:10.429314    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:10.429450 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:10.429393    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls podName:05376a4a-6b6c-46cf-b8aa-77f1e8995a27 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:11.429372279 +0000 UTC m=+35.720486134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls") pod "dns-default-m4qxw" (UID: "05376a4a-6b6c-46cf-b8aa-77f1e8995a27") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:11.336261 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:11.336221    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:11.336630 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:11.336379    2537 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:07:11.336630 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:11.336398    2537 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648b8d4db-5pgz2: secret "image-registry-tls" not found
Sep 17 14:07:11.336630 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:11.336454    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls podName:1d4ece8e-8652-48fc-bd1e-77005ef9b11d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:13.336436837 +0000 UTC m=+37.627550675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls") pod "image-registry-648b8d4db-5pgz2" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d") : secret "image-registry-tls" not found
Sep 17 14:07:11.437091 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:11.437067    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:11.437268 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:11.437246    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:11.437341 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:11.437330    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls podName:05376a4a-6b6c-46cf-b8aa-77f1e8995a27 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:13.437307729 +0000 UTC m=+37.728421592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls") pod "dns-default-m4qxw" (UID: "05376a4a-6b6c-46cf-b8aa-77f1e8995a27") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:11.594101 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:11.594039    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zsfks"]
Sep 17 14:07:11.606004 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:07:11.605980    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916ac6c2_3233_45cd_9c79_088264c87d00.slice/crio-2b3b660d63a6ccabd2cde7a5590926f567a014689bc8e1dc768b8b912bd4e590 WatchSource:0}: Error finding container 2b3b660d63a6ccabd2cde7a5590926f567a014689bc8e1dc768b8b912bd4e590: Status 404 returned error can't find the container with id 2b3b660d63a6ccabd2cde7a5590926f567a014689bc8e1dc768b8b912bd4e590
Sep 17 14:07:12.453926 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:12.453891    2537 generic.go:356] "Generic (PLEG): container finished" podID="409668f2-1424-47ba-954c-39a313d33e54" containerID="4e5b560676e100695fdda9ea4067cb1240c90abead243ad9a5b17bc23d081c61" exitCode=0
Sep 17 14:07:12.454894 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:12.453956    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8kj2" event={"ID":"409668f2-1424-47ba-954c-39a313d33e54","Type":"ContainerDied","Data":"4e5b560676e100695fdda9ea4067cb1240c90abead243ad9a5b17bc23d081c61"}
Sep 17 14:07:12.455311 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:12.455285    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zsfks" event={"ID":"916ac6c2-3233-45cd-9c79-088264c87d00","Type":"ContainerStarted","Data":"2b3b660d63a6ccabd2cde7a5590926f567a014689bc8e1dc768b8b912bd4e590"}
Sep 17 14:07:13.348161 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:13.347915    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:13.348369 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:13.348082    2537 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:07:13.348369 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:13.348204    2537 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648b8d4db-5pgz2: secret "image-registry-tls" not found
Sep 17 14:07:13.348369 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:13.348265    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls podName:1d4ece8e-8652-48fc-bd1e-77005ef9b11d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:17.348249745 +0000 UTC m=+41.639363585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls") pod "image-registry-648b8d4db-5pgz2" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d") : secret "image-registry-tls" not found
Sep 17 14:07:13.449015 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:13.448982    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:13.449212 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:13.449160    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:13.449279 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:13.449247    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls podName:05376a4a-6b6c-46cf-b8aa-77f1e8995a27 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:17.449228394 +0000 UTC m=+41.740342265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls") pod "dns-default-m4qxw" (UID: "05376a4a-6b6c-46cf-b8aa-77f1e8995a27") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:13.461718 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:13.461685    2537 generic.go:356] "Generic (PLEG): container finished" podID="409668f2-1424-47ba-954c-39a313d33e54" containerID="772550a9f9ef7946246e9af92ca083361f5cef08bd5bfb9f85228309537e64e9" exitCode=0
Sep 17 14:07:13.462127 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:13.461741    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8kj2" event={"ID":"409668f2-1424-47ba-954c-39a313d33e54","Type":"ContainerDied","Data":"772550a9f9ef7946246e9af92ca083361f5cef08bd5bfb9f85228309537e64e9"}
Sep 17 14:07:14.331439 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:14.331406    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h78nv_d1d14525-b7fc-4b75-bae7-a22fbca657f3/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:14.423193 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:14.423140    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h78nv_d1d14525-b7fc-4b75-bae7-a22fbca657f3/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:14.467841 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:14.467811    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8kj2" event={"ID":"409668f2-1424-47ba-954c-39a313d33e54","Type":"ContainerStarted","Data":"07f3af0ca32d3c2f6a738f12beed2eed08135ed913e7711f26ddc059e9b7dfd0"}
Sep 17 14:07:14.469379 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:14.469357    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zsfks" event={"ID":"916ac6c2-3233-45cd-9c79-088264c87d00","Type":"ContainerStarted","Data":"a53acd4f6f6d0641053fdf77c1cdab39441e1ef5b82ac024c2d5b9649fb08a95"}
Sep 17 14:07:14.501221 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:14.501146    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x8kj2" podStartSLOduration=10.109646007 podStartE2EDuration="38.501127519s" totalImagesPullingTime="28.391481512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:14.498805432 +0000 UTC m=+38.789919322" watchObservedRunningTime="2026-09-17 14:07:14.501127519 +0000 UTC m=+38.792241381"
Sep 17 14:07:14.533390 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:14.533287    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zsfks" podStartSLOduration=2.887163758 podStartE2EDuration="5.533270775s" totalImagesPullingTime="2.646107017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:14.531838664 +0000 UTC m=+38.822952525" watchObservedRunningTime="2026-09-17 14:07:14.533270775 +0000 UTC m=+38.824384643"
Sep 17 14:07:14.534304 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:14.534263    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h78nv_d1d14525-b7fc-4b75-bae7-a22fbca657f3/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:14.716388 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:14.716350    2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h78nv_d1d14525-b7fc-4b75-bae7-a22fbca657f3/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:17.375121 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:17.375084    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:17.375518 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:17.375221    2537 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:07:17.375518 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:17.375233    2537 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648b8d4db-5pgz2: secret "image-registry-tls" not found
Sep 17 14:07:17.375518 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:17.375280    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls podName:1d4ece8e-8652-48fc-bd1e-77005ef9b11d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:25.375265646 +0000 UTC m=+49.666379485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls") pod "image-registry-648b8d4db-5pgz2" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d") : secret "image-registry-tls" not found
Sep 17 14:07:17.475405 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:17.475376    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:17.475605 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:17.475525    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:17.475605 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:17.475587    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls podName:05376a4a-6b6c-46cf-b8aa-77f1e8995a27 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:25.475568262 +0000 UTC m=+49.766682104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls") pod "dns-default-m4qxw" (UID: "05376a4a-6b6c-46cf-b8aa-77f1e8995a27") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:22.010055 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.009976    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm"]
Sep 17 14:07:22.048948 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.048914    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"]
Sep 17 14:07:22.049131 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.049080    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm"
Sep 17 14:07:22.053763 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.053736    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 14:07:22.053763 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.053751    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ls99d\""
Sep 17 14:07:22.053967 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.053736    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 14:07:22.053967 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.053739    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 14:07:22.053967 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.053786    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 14:07:22.064674 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.064635    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:22.064816 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.064646    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"]
Sep 17 14:07:22.071681 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.071655    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 14:07:22.086648 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.086616    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm"]
Sep 17 14:07:22.086648 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.086655    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"]
Sep 17 14:07:22.086837 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.086683    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"]
Sep 17 14:07:22.086837 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.086737    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.090310 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.090286    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 14:07:22.090587 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.090568    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 14:07:22.090705 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.090582    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 14:07:22.090705 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.090655    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 14:07:22.210797 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.210757    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/db88b394-d715-41fc-80a1-bc2257b06186-klusterlet-config\") pod \"klusterlet-addon-workmgr-77c97794cc-7nssj\" (UID: \"db88b394-d715-41fc-80a1-bc2257b06186\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:22.210797 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.210795    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.211001 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.210826    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-ca\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.211001 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.210870    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.211001 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.210902    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmv4v\" (UniqueName: \"kubernetes.io/projected/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-kube-api-access-mmv4v\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.211001 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.210921    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw6tc\" (UniqueName: \"kubernetes.io/projected/db88b394-d715-41fc-80a1-bc2257b06186-kube-api-access-cw6tc\") pod \"klusterlet-addon-workmgr-77c97794cc-7nssj\" (UID: \"db88b394-d715-41fc-80a1-bc2257b06186\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:22.211001 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.210966    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9rzt\" (UniqueName: \"kubernetes.io/projected/f135e91d-d078-49d3-aca0-1cea8db0596f-kube-api-access-l9rzt\") pod \"managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm\" (UID: \"f135e91d-d078-49d3-aca0-1cea8db0596f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm"
Sep 17 14:07:22.211001 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.210999    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.211256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.211092    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f135e91d-d078-49d3-aca0-1cea8db0596f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm\" (UID: \"f135e91d-d078-49d3-aca0-1cea8db0596f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm"
Sep 17 14:07:22.211256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.211119    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db88b394-d715-41fc-80a1-bc2257b06186-tmp\") pod \"klusterlet-addon-workmgr-77c97794cc-7nssj\" (UID: \"db88b394-d715-41fc-80a1-bc2257b06186\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:22.211256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.211144    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-hub\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.311579 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311504    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f135e91d-d078-49d3-aca0-1cea8db0596f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm\" (UID: \"f135e91d-d078-49d3-aca0-1cea8db0596f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm"
Sep 17 14:07:22.311579 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311537    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db88b394-d715-41fc-80a1-bc2257b06186-tmp\") pod \"klusterlet-addon-workmgr-77c97794cc-7nssj\" (UID: \"db88b394-d715-41fc-80a1-bc2257b06186\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:22.311579 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311563    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-hub\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.311579 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311577    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/db88b394-d715-41fc-80a1-bc2257b06186-klusterlet-config\") pod \"klusterlet-addon-workmgr-77c97794cc-7nssj\" (UID: \"db88b394-d715-41fc-80a1-bc2257b06186\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:22.311878 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311595    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.311878 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311675    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-ca\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.311878 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311706    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.311878 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311734    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmv4v\" (UniqueName: \"kubernetes.io/projected/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-kube-api-access-mmv4v\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.311878 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311757    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6tc\" (UniqueName: \"kubernetes.io/projected/db88b394-d715-41fc-80a1-bc2257b06186-kube-api-access-cw6tc\") pod \"klusterlet-addon-workmgr-77c97794cc-7nssj\" (UID: \"db88b394-d715-41fc-80a1-bc2257b06186\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:22.311878 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311799    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9rzt\" (UniqueName: \"kubernetes.io/projected/f135e91d-d078-49d3-aca0-1cea8db0596f-kube-api-access-l9rzt\") pod \"managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm\" (UID: \"f135e91d-d078-49d3-aca0-1cea8db0596f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm"
Sep 17 14:07:22.311878 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311846    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.312271 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.311957    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db88b394-d715-41fc-80a1-bc2257b06186-tmp\") pod \"klusterlet-addon-workmgr-77c97794cc-7nssj\" (UID: \"db88b394-d715-41fc-80a1-bc2257b06186\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:22.312650 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.312622    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.316040 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.316010    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-hub\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.316040 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.316033    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.316245 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.316223    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f135e91d-d078-49d3-aca0-1cea8db0596f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm\" (UID: \"f135e91d-d078-49d3-aca0-1cea8db0596f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm"
Sep 17 14:07:22.316519 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.316500    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-ca\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.316594 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.316578    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/db88b394-d715-41fc-80a1-bc2257b06186-klusterlet-config\") pod \"klusterlet-addon-workmgr-77c97794cc-7nssj\" (UID: \"db88b394-d715-41fc-80a1-bc2257b06186\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:22.316594 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.316586    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.321539 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.321518    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9rzt\" (UniqueName: \"kubernetes.io/projected/f135e91d-d078-49d3-aca0-1cea8db0596f-kube-api-access-l9rzt\") pod \"managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm\" (UID: \"f135e91d-d078-49d3-aca0-1cea8db0596f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm"
Sep 17 14:07:22.325282 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.325250    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmv4v\" (UniqueName: \"kubernetes.io/projected/e4f49a03-125a-4a69-ba9d-c6d18e6bc401-kube-api-access-mmv4v\") pod \"cluster-proxy-proxy-agent-58c5cc9676-57b7n\" (UID: \"e4f49a03-125a-4a69-ba9d-c6d18e6bc401\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.325379 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.325354    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6tc\" (UniqueName: \"kubernetes.io/projected/db88b394-d715-41fc-80a1-bc2257b06186-kube-api-access-cw6tc\") pod \"klusterlet-addon-workmgr-77c97794cc-7nssj\" (UID: \"db88b394-d715-41fc-80a1-bc2257b06186\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:22.369406 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.369365    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm"
Sep 17 14:07:22.377608 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.377579    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:22.395825 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.395790    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:07:22.702390 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.702358    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm"]
Sep 17 14:07:22.705522 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:07:22.705498    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf135e91d_d078_49d3_aca0_1cea8db0596f.slice/crio-53df92cfe6b8b20fba4b1aa6c6cef662dff42cb559906528c2850aadfbfcf11c WatchSource:0}: Error finding container 53df92cfe6b8b20fba4b1aa6c6cef662dff42cb559906528c2850aadfbfcf11c: Status 404 returned error can't find the container with id 53df92cfe6b8b20fba4b1aa6c6cef662dff42cb559906528c2850aadfbfcf11c
Sep 17 14:07:22.723200 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.723150    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"]
Sep 17 14:07:22.725400 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:07:22.725375    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb88b394_d715_41fc_80a1_bc2257b06186.slice/crio-62de5f5611f62a1d09cd31c767a6fbac8d03ef3be3a799069e220e0a7b81cdee WatchSource:0}: Error finding container 62de5f5611f62a1d09cd31c767a6fbac8d03ef3be3a799069e220e0a7b81cdee: Status 404 returned error can't find the container with id 62de5f5611f62a1d09cd31c767a6fbac8d03ef3be3a799069e220e0a7b81cdee
Sep 17 14:07:22.757150 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:22.757106    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"]
Sep 17 14:07:22.759579 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:07:22.759556    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4f49a03_125a_4a69_ba9d_c6d18e6bc401.slice/crio-2b9c582a61d4f0630a2290de5e905ebb22951ba190289b717280b37f80fc64f1 WatchSource:0}: Error finding container 2b9c582a61d4f0630a2290de5e905ebb22951ba190289b717280b37f80fc64f1: Status 404 returned error can't find the container with id 2b9c582a61d4f0630a2290de5e905ebb22951ba190289b717280b37f80fc64f1
Sep 17 14:07:23.493339 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:23.493258    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" event={"ID":"e4f49a03-125a-4a69-ba9d-c6d18e6bc401","Type":"ContainerStarted","Data":"2b9c582a61d4f0630a2290de5e905ebb22951ba190289b717280b37f80fc64f1"}
Sep 17 14:07:23.495539 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:23.495504    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj" event={"ID":"db88b394-d715-41fc-80a1-bc2257b06186","Type":"ContainerStarted","Data":"62de5f5611f62a1d09cd31c767a6fbac8d03ef3be3a799069e220e0a7b81cdee"}
Sep 17 14:07:23.497084 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:23.497056    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm" event={"ID":"f135e91d-d078-49d3-aca0-1cea8db0596f","Type":"ContainerStarted","Data":"53df92cfe6b8b20fba4b1aa6c6cef662dff42cb559906528c2850aadfbfcf11c"}
Sep 17 14:07:25.439965 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:25.439651    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:25.439965 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:25.439823    2537 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:07:25.439965 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:25.439838    2537 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648b8d4db-5pgz2: secret "image-registry-tls" not found
Sep 17 14:07:25.439965 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:25.439893    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls podName:1d4ece8e-8652-48fc-bd1e-77005ef9b11d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:41.439876744 +0000 UTC m=+65.730990588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls") pod "image-registry-648b8d4db-5pgz2" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d") : secret "image-registry-tls" not found
Sep 17 14:07:25.540250 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:25.540092    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:25.540435 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:25.540330    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:25.540435 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:25.540396    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls podName:05376a4a-6b6c-46cf-b8aa-77f1e8995a27 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:41.54037862 +0000 UTC m=+65.831492473 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls") pod "dns-default-m4qxw" (UID: "05376a4a-6b6c-46cf-b8aa-77f1e8995a27") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:30.521194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:30.521151    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" event={"ID":"e4f49a03-125a-4a69-ba9d-c6d18e6bc401","Type":"ContainerStarted","Data":"d5ab742920dfaf3f79972709fde4e0d155248a4b3ddac15a6109b19c385e801f"}
Sep 17 14:07:30.521660 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:30.521211    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" event={"ID":"e4f49a03-125a-4a69-ba9d-c6d18e6bc401","Type":"ContainerStarted","Data":"db87d61a174105d5bc8ab62cb72f9d0f41dd99847e46726228d9f08f5ee5f580"}
Sep 17 14:07:30.521660 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:30.521228    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" event={"ID":"e4f49a03-125a-4a69-ba9d-c6d18e6bc401","Type":"ContainerStarted","Data":"ff01093361ab45a8722529a241437e1426e9251ef67c14412da50f2be3c9fd54"}
Sep 17 14:07:30.522595 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:30.522563    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj" event={"ID":"db88b394-d715-41fc-80a1-bc2257b06186","Type":"ContainerStarted","Data":"fa86eb09802999b147f0a885ad8b6ec3c7cd5e3b96837c2d41ab9b75e137103f"}
Sep 17 14:07:30.522785 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:30.522760    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:30.524119 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:30.524094    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm" event={"ID":"f135e91d-d078-49d3-aca0-1cea8db0596f","Type":"ContainerStarted","Data":"4f3f55e63353f845aff13fafd482bb067fe5c38dcfc115c5de318161e97003c1"}
Sep 17 14:07:30.524566 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:30.524551    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:07:30.566389 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:30.566345    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" podStartSLOduration=1.63270631 podStartE2EDuration="8.566332228s" totalImagesPullingTime="6.933625918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:30.56481475 +0000 UTC m=+54.855928612" watchObservedRunningTime="2026-09-17 14:07:30.566332228 +0000 UTC m=+54.857446080"
Sep 17 14:07:30.635825 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:30.635779    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm" podStartSLOduration=2.638744119 podStartE2EDuration="9.635763551s" totalImagesPullingTime="6.997019432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:30.603480331 +0000 UTC m=+54.894594191" watchObservedRunningTime="2026-09-17 14:07:30.635763551 +0000 UTC m=+54.926877423"
Sep 17 14:07:30.635966 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:30.635848    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj" podStartSLOduration=2.587569353 podStartE2EDuration="9.63584458s" totalImagesPullingTime="7.048275227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:30.634917609 +0000 UTC m=+54.926031470" watchObservedRunningTime="2026-09-17 14:07:30.63584458 +0000 UTC m=+54.926958440"
Sep 17 14:07:35.451051 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:35.451021    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fn7m"
Sep 17 14:07:41.463817 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:41.463774    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:07:41.464240 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:41.463916    2537 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:07:41.464240 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:41.463928    2537 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648b8d4db-5pgz2: secret "image-registry-tls" not found
Sep 17 14:07:41.464240 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:41.463976    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls podName:1d4ece8e-8652-48fc-bd1e-77005ef9b11d nodeName:}" failed. No retries permitted until 2026-09-17 14:08:13.463963821 +0000 UTC m=+97.755077673 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls") pod "image-registry-648b8d4db-5pgz2" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d") : secret "image-registry-tls" not found
Sep 17 14:07:41.565137 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:41.565109    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:07:41.565285 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:41.565238    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:41.565327 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:41.565286    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls podName:05376a4a-6b6c-46cf-b8aa-77f1e8995a27 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:13.565273966 +0000 UTC m=+97.856387806 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls") pod "dns-default-m4qxw" (UID: "05376a4a-6b6c-46cf-b8aa-77f1e8995a27") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:41.867651 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:41.867620    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:07:41.871063 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:41.871041    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 14:07:41.878017 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:41.878000    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 14:07:41.878072 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:07:41.878062    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs podName:d2d8f52c-5035-471e-b6b0-c95e222670a8 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:45.878041526 +0000 UTC m=+130.169155368 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs") pod "network-metrics-daemon-kp9pm" (UID: "d2d8f52c-5035-471e-b6b0-c95e222670a8") : secret "metrics-daemon-secret" not found
Sep 17 14:07:41.968859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:41.968823    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:07:41.972275 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:41.972258    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 14:07:41.981906 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:41.981877    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16a9d26e-dfaa-4330-be84-a026c41d6f03-original-pull-secret\") pod \"global-pull-secret-syncer-8dc5c\" (UID: \"16a9d26e-dfaa-4330-be84-a026c41d6f03\") " pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:07:42.065320 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:42.065293    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dc5c"
Sep 17 14:07:42.069643 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:42.069621    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngxf\" (UniqueName: \"kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf\") pod \"network-check-target-kkfch\" (UID: \"0dff3702-7b53-480c-ae7a-074f122eb7e8\") " pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:42.073170 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:42.073153    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 14:07:42.083743 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:42.083726    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 14:07:42.092823 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:42.092798    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ngxf\" (UniqueName: \"kubernetes.io/projected/0dff3702-7b53-480c-ae7a-074f122eb7e8-kube-api-access-5ngxf\") pod \"network-check-target-kkfch\" (UID: \"0dff3702-7b53-480c-ae7a-074f122eb7e8\") " pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:42.357387 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:42.357273    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8dc5c"]
Sep 17 14:07:42.360034 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:07:42.360007    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a9d26e_dfaa_4330_be84_a026c41d6f03.slice/crio-78c1ba6ebc46d0a3095beedc7ddf9635028fead786239e145c7deda329e9c24c WatchSource:0}: Error finding container 78c1ba6ebc46d0a3095beedc7ddf9635028fead786239e145c7deda329e9c24c: Status 404 returned error can't find the container with id 78c1ba6ebc46d0a3095beedc7ddf9635028fead786239e145c7deda329e9c24c
Sep 17 14:07:42.362036 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:42.362019    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c2694\""
Sep 17 14:07:42.370075 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:42.370053    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:42.562334 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:42.562301    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8dc5c" event={"ID":"16a9d26e-dfaa-4330-be84-a026c41d6f03","Type":"ContainerStarted","Data":"78c1ba6ebc46d0a3095beedc7ddf9635028fead786239e145c7deda329e9c24c"}
Sep 17 14:07:42.662043 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:42.662018    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kkfch"]
Sep 17 14:07:42.664075 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:07:42.664049    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dff3702_7b53_480c_ae7a_074f122eb7e8.slice/crio-d9bd45541ad8278c50c78ad7c36a4300636b54d3a718226ce33ce0c1bdbc4814 WatchSource:0}: Error finding container d9bd45541ad8278c50c78ad7c36a4300636b54d3a718226ce33ce0c1bdbc4814: Status 404 returned error can't find the container with id d9bd45541ad8278c50c78ad7c36a4300636b54d3a718226ce33ce0c1bdbc4814
Sep 17 14:07:43.568441 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:43.568403    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kkfch" event={"ID":"0dff3702-7b53-480c-ae7a-074f122eb7e8","Type":"ContainerStarted","Data":"d9bd45541ad8278c50c78ad7c36a4300636b54d3a718226ce33ce0c1bdbc4814"}
Sep 17 14:07:47.583735 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:47.583698    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kkfch" event={"ID":"0dff3702-7b53-480c-ae7a-074f122eb7e8","Type":"ContainerStarted","Data":"af07b4b505b66fd862b96f5d0dea54b6a5f35e83672d58d996e16103d66775e9"}
Sep 17 14:07:47.584257 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:47.583839    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:07:47.585242 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:47.585210    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8dc5c" event={"ID":"16a9d26e-dfaa-4330-be84-a026c41d6f03","Type":"ContainerStarted","Data":"c4dde1dc993432d30501ddc27202ee2341c18d5b259e518f9f8d1eb73d11e86d"}
Sep 17 14:07:47.602866 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:47.602799    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kkfch" podStartSLOduration=66.869571179 podStartE2EDuration="1m11.602779187s" totalImagesPullingTime="4.733208008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:47.601761847 +0000 UTC m=+71.892875709" watchObservedRunningTime="2026-09-17 14:07:47.602779187 +0000 UTC m=+71.893893040"
Sep 17 14:07:47.617767 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:07:47.617702    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8dc5c" podStartSLOduration=65.580611794 podStartE2EDuration="1m10.617680836s" totalImagesPullingTime="5.037069042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:47.617202173 +0000 UTC m=+71.908316035" watchObservedRunningTime="2026-09-17 14:07:47.617680836 +0000 UTC m=+71.908794698"
Sep 17 14:08:13.495619 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:13.495495    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:08:13.495619 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:13.495613    2537 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:08:13.496096 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:13.495624    2537 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648b8d4db-5pgz2: secret "image-registry-tls" not found
Sep 17 14:08:13.496096 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:13.495673    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls podName:1d4ece8e-8652-48fc-bd1e-77005ef9b11d nodeName:}" failed. No retries permitted until 2026-09-17 14:09:17.495658978 +0000 UTC m=+161.786772816 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls") pod "image-registry-648b8d4db-5pgz2" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d") : secret "image-registry-tls" not found
Sep 17 14:08:13.596196 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:13.596136    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:08:13.596343 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:13.596289    2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:08:13.596382 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:13.596364    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls podName:05376a4a-6b6c-46cf-b8aa-77f1e8995a27 nodeName:}" failed. No retries permitted until 2026-09-17 14:09:17.59634864 +0000 UTC m=+161.887462483 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls") pod "dns-default-m4qxw" (UID: "05376a4a-6b6c-46cf-b8aa-77f1e8995a27") : secret "dns-default-metrics-tls" not found
Sep 17 14:08:18.592035 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:18.591999    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kkfch"
Sep 17 14:08:45.930651 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:45.930598    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:08:45.931135 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:45.930766    2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 14:08:45.931135 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:45.930841    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs podName:d2d8f52c-5035-471e-b6b0-c95e222670a8 nodeName:}" failed. No retries permitted until 2026-09-17 14:10:47.930826828 +0000 UTC m=+252.221940667 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs") pod "network-metrics-daemon-kp9pm" (UID: "d2d8f52c-5035-471e-b6b0-c95e222670a8") : secret "metrics-daemon-secret" not found
Sep 17 14:08:49.293102 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.293068    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d7v4c"]
Sep 17 14:08:49.296246 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.296230    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.299980 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.299950    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 14:08:49.300114 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.300023    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 14:08:49.301068 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.301054    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 14:08:49.301130 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.301055    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tmmrx\""
Sep 17 14:08:49.301130 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.301116    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 14:08:49.345302 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.345264    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7v4c"]
Sep 17 14:08:49.455142 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.455110    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.455334 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.455163    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.455334 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.455306    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcshh\" (UniqueName: \"kubernetes.io/projected/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-api-access-hcshh\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.455406 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.455338    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-crio-socket\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.455406 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.455360    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-data-volume\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.556643 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.556560    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-crio-socket\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.556643 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.556591    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-data-volume\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.556643 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.556613    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.556917 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.556649    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.556917 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.556688    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-crio-socket\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.556917 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.556695    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcshh\" (UniqueName: \"kubernetes.io/projected/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-api-access-hcshh\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.556917 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:49.556716    2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:08:49.556917 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:49.556778    2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:49.556917 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:49.556802    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm podName:6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:50.056782334 +0000 UTC m=+134.347896189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7v4c" (UID: "6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5") : configmap "kube-rbac-proxy" not found
Sep 17 14:08:49.556917 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:49.556834    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls podName:6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:50.056822752 +0000 UTC m=+134.347936594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7v4c" (UID: "6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:49.557204 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.556924    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-data-volume\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:49.584279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:49.584248    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcshh\" (UniqueName: \"kubernetes.io/projected/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-api-access-hcshh\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:50.060799 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:50.060762    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:50.060968 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:50.060813    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:50.060968 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:50.060883    2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:08:50.060968 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:50.060919    2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:50.060968 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:50.060958    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm podName:6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:51.060942541 +0000 UTC m=+135.352056379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7v4c" (UID: "6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5") : configmap "kube-rbac-proxy" not found
Sep 17 14:08:50.061107 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:50.060972    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls podName:6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:51.060965795 +0000 UTC m=+135.352079634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7v4c" (UID: "6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:51.031682 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:51.031658    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8fgsv_d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03/dns-node-resolver/0.log"
Sep 17 14:08:51.068102 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:51.068068    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:51.068308 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:51.068121    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:51.068308 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:51.068235    2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:08:51.068420 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:51.068312    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm podName:6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:53.068292238 +0000 UTC m=+137.359406095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7v4c" (UID: "6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5") : configmap "kube-rbac-proxy" not found
Sep 17 14:08:51.068420 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:51.068352    2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:51.068420 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:51.068394    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls podName:6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:53.068383673 +0000 UTC m=+137.359497512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7v4c" (UID: "6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:52.432599 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:52.432573    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xz96l_f856d63c-e51d-4532-a4e1-73fc32c4c203/node-ca/0.log"
Sep 17 14:08:53.086263 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:53.086230    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:53.086446 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:53.086291    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:53.086446 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:53.086362    2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:08:53.086446 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:53.086434    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm podName:6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:57.086414341 +0000 UTC m=+141.377528202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7v4c" (UID: "6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5") : configmap "kube-rbac-proxy" not found
Sep 17 14:08:53.086446 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:53.086440    2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:53.086595 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:53.086474    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls podName:6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:57.08646414 +0000 UTC m=+141.377577979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7v4c" (UID: "6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:53.239580 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:53.239533    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zsfks_916ac6c2-3233-45cd-9c79-088264c87d00/serve-healthcheck-canary/0.log"
Sep 17 14:08:57.116599 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:57.116556    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:57.117044 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:08:57.116608    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:08:57.117044 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:57.116686    2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:08:57.117044 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:57.116709    2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:57.117044 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:57.116760    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls podName:6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5 nodeName:}" failed. No retries permitted until 2026-09-17 14:09:05.116746615 +0000 UTC m=+149.407860454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7v4c" (UID: "6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:57.117044 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:08:57.116776    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm podName:6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5 nodeName:}" failed. No retries permitted until 2026-09-17 14:09:05.116766772 +0000 UTC m=+149.407880611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7v4c" (UID: "6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5") : configmap "kube-rbac-proxy" not found
Sep 17 14:09:02.397508 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:02.397462    2537 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" podUID="e4f49a03-125a-4a69-ba9d-c6d18e6bc401" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 14:09:05.183296 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:05.183245    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:09:05.183296 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:05.183313    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:09:05.183903 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:05.183877    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:09:05.185592 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:05.185574    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7v4c\" (UID: \"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5\") " pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:09:05.204655 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:05.204611    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7v4c"
Sep 17 14:09:05.477787 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:05.477696    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7v4c"]
Sep 17 14:09:05.481472 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:09:05.481444    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a66b70e_4c3a_4a3c_b374_d60c2e06cdd5.slice/crio-6bd8e22f550451cf639da0e48c4278c49f0cea32e75a04e73fd13e14e80773ed WatchSource:0}: Error finding container 6bd8e22f550451cf639da0e48c4278c49f0cea32e75a04e73fd13e14e80773ed: Status 404 returned error can't find the container with id 6bd8e22f550451cf639da0e48c4278c49f0cea32e75a04e73fd13e14e80773ed
Sep 17 14:09:05.813750 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:05.813668    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7v4c" event={"ID":"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5","Type":"ContainerStarted","Data":"91af9e534240ed8fd1cb2ecc12891e2144db5cd668e997b34b5c4d5c9569b4fe"}
Sep 17 14:09:05.813750 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:05.813724    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7v4c" event={"ID":"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5","Type":"ContainerStarted","Data":"6bd8e22f550451cf639da0e48c4278c49f0cea32e75a04e73fd13e14e80773ed"}
Sep 17 14:09:06.818929 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:06.818897    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7v4c" event={"ID":"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5","Type":"ContainerStarted","Data":"a5588debd025c1b00eb3dac3db6385c086e933070c2b7b8b56d78e9891564382"}
Sep 17 14:09:07.823851 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:07.823757    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7v4c" event={"ID":"6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5","Type":"ContainerStarted","Data":"3c249d90e1a6d42e3b5b4cab508ec16eaedcb55694ea7d6cea765175e1b92875"}
Sep 17 14:09:07.850437 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:07.850390    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d7v4c" podStartSLOduration=16.993621275 podStartE2EDuration="18.850375748s" totalImagesPullingTime="1.856754473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:08:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:07.848074031 +0000 UTC m=+152.139187883" watchObservedRunningTime="2026-09-17 14:09:07.850375748 +0000 UTC m=+152.141489634"
Sep 17 14:09:12.397588 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:12.397551    2537 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" podUID="e4f49a03-125a-4a69-ba9d-c6d18e6bc401" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 14:09:12.613442 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:09:12.613397    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2" podUID="1d4ece8e-8652-48fc-bd1e-77005ef9b11d"
Sep 17 14:09:12.770950 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:09:12.770841    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-m4qxw" podUID="05376a4a-6b6c-46cf-b8aa-77f1e8995a27"
Sep 17 14:09:12.838859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:12.838824    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:09:12.839021 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:12.838824    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:09:13.252263 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:09:13.252217    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kp9pm" podUID="d2d8f52c-5035-471e-b6b0-c95e222670a8"
Sep 17 14:09:17.572696 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:17.572652    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:09:17.574904 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:17.574884    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"image-registry-648b8d4db-5pgz2\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") " pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:09:17.643335 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:17.643308    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2g4kt\""
Sep 17 14:09:17.650705 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:17.650688    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:09:17.673751 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:17.673728    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:09:17.676324 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:17.676305    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05376a4a-6b6c-46cf-b8aa-77f1e8995a27-metrics-tls\") pod \"dns-default-m4qxw\" (UID: \"05376a4a-6b6c-46cf-b8aa-77f1e8995a27\") " pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:09:17.924444 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:17.924412    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-648b8d4db-5pgz2"]
Sep 17 14:09:17.928713 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:09:17.928679    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4ece8e_8652_48fc_bd1e_77005ef9b11d.slice/crio-54bd23ff731d5d229510d560f712089076582d6c64072c0fd6f98a4ddcfc1660 WatchSource:0}: Error finding container 54bd23ff731d5d229510d560f712089076582d6c64072c0fd6f98a4ddcfc1660: Status 404 returned error can't find the container with id 54bd23ff731d5d229510d560f712089076582d6c64072c0fd6f98a4ddcfc1660
Sep 17 14:09:17.941988 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:17.941970    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jdh9k\""
Sep 17 14:09:17.950051 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:17.950032    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:09:18.230677 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:18.230655    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m4qxw"]
Sep 17 14:09:18.233561 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:09:18.233533    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05376a4a_6b6c_46cf_b8aa_77f1e8995a27.slice/crio-3a0787ffde12005e99317c7bd4e28b04afeff579fc4671662111c8f892027a78 WatchSource:0}: Error finding container 3a0787ffde12005e99317c7bd4e28b04afeff579fc4671662111c8f892027a78: Status 404 returned error can't find the container with id 3a0787ffde12005e99317c7bd4e28b04afeff579fc4671662111c8f892027a78
Sep 17 14:09:18.858496 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:18.858459    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2" event={"ID":"1d4ece8e-8652-48fc-bd1e-77005ef9b11d","Type":"ContainerStarted","Data":"9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98"}
Sep 17 14:09:18.858915 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:18.858510    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2" event={"ID":"1d4ece8e-8652-48fc-bd1e-77005ef9b11d","Type":"ContainerStarted","Data":"54bd23ff731d5d229510d560f712089076582d6c64072c0fd6f98a4ddcfc1660"}
Sep 17 14:09:18.858915 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:18.858568    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:09:18.859657 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:18.859632    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4qxw" event={"ID":"05376a4a-6b6c-46cf-b8aa-77f1e8995a27","Type":"ContainerStarted","Data":"3a0787ffde12005e99317c7bd4e28b04afeff579fc4671662111c8f892027a78"}
Sep 17 14:09:18.890871 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:18.890829    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2" podStartSLOduration=161.890816689 podStartE2EDuration="2m41.890816689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:18.88904101 +0000 UTC m=+163.180154884" watchObservedRunningTime="2026-09-17 14:09:18.890816689 +0000 UTC m=+163.181930551"
Sep 17 14:09:20.867762 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:20.867722    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4qxw" event={"ID":"05376a4a-6b6c-46cf-b8aa-77f1e8995a27","Type":"ContainerStarted","Data":"56169e9c01d08780e1d9d432809fef69c16e73a5bb8d496c1b67d70b065696a0"}
Sep 17 14:09:20.868192 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:20.867779    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4qxw" event={"ID":"05376a4a-6b6c-46cf-b8aa-77f1e8995a27","Type":"ContainerStarted","Data":"f9894304b613be61383e3e7953394b99c42396c9762187bc425a3aa891f7ac00"}
Sep 17 14:09:20.868192 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:20.867826    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:09:20.888869 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:20.888823    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m4qxw" podStartSLOduration=129.896137916 podStartE2EDuration="2m11.888807468s" totalImagesPullingTime="1.992669552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:20.887095775 +0000 UTC m=+165.178209636" watchObservedRunningTime="2026-09-17 14:09:20.888807468 +0000 UTC m=+165.179921363"
Sep 17 14:09:22.397170 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:22.397129    2537 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" podUID="e4f49a03-125a-4a69-ba9d-c6d18e6bc401" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 14:09:22.397616 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:22.397211    2537 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n"
Sep 17 14:09:22.397746 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:22.397714    2537 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"d5ab742920dfaf3f79972709fde4e0d155248a4b3ddac15a6109b19c385e801f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 14:09:22.397785 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:22.397770    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" podUID="e4f49a03-125a-4a69-ba9d-c6d18e6bc401" containerName="service-proxy" containerID="cri-o://d5ab742920dfaf3f79972709fde4e0d155248a4b3ddac15a6109b19c385e801f" gracePeriod=30
Sep 17 14:09:22.875698 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:22.875667    2537 generic.go:356] "Generic (PLEG): container finished" podID="e4f49a03-125a-4a69-ba9d-c6d18e6bc401" containerID="d5ab742920dfaf3f79972709fde4e0d155248a4b3ddac15a6109b19c385e801f" exitCode=2
Sep 17 14:09:22.875894 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:22.875741    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" event={"ID":"e4f49a03-125a-4a69-ba9d-c6d18e6bc401","Type":"ContainerDied","Data":"d5ab742920dfaf3f79972709fde4e0d155248a4b3ddac15a6109b19c385e801f"}
Sep 17 14:09:22.875894 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:22.875798    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58c5cc9676-57b7n" event={"ID":"e4f49a03-125a-4a69-ba9d-c6d18e6bc401","Type":"ContainerStarted","Data":"c66d127f64eceea9564cc14b902697475912c1f5d86f8a1db4f539efb4386a2a"}
Sep 17 14:09:24.013796 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.013762    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bwhn7"]
Sep 17 14:09:24.018609 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.018592    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.023072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.023044    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 14:09:24.023072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.023044    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rvp8v\""
Sep 17 14:09:24.024254 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.024222    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 14:09:24.024254 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.024239    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 14:09:24.024441 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.024278    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 14:09:24.024441 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.024303    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 14:09:24.024608 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.024577    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 14:09:24.122111 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.122073    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-accelerators-collector-config\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.122111 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.122114    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/edd8cf01-23e4-4cca-bce4-52210d98b048-root\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.122336 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.122135    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edd8cf01-23e4-4cca-bce4-52210d98b048-sys\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.122336 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.122274    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-tls\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.122336 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.122301    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-textfile\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.122336 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.122318    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.122336 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.122337    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-wtmp\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.122561 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.122420    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zr9f\" (UniqueName: \"kubernetes.io/projected/edd8cf01-23e4-4cca-bce4-52210d98b048-kube-api-access-9zr9f\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.122561 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.122458    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edd8cf01-23e4-4cca-bce4-52210d98b048-metrics-client-ca\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.223594 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223559    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-tls\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.223594 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223598    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-textfile\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.223799 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223617    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.223799 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223635    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-wtmp\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.223799 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223651    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zr9f\" (UniqueName: \"kubernetes.io/projected/edd8cf01-23e4-4cca-bce4-52210d98b048-kube-api-access-9zr9f\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.223799 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223675    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edd8cf01-23e4-4cca-bce4-52210d98b048-metrics-client-ca\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.223799 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223718    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-accelerators-collector-config\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.223799 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:09:24.223721    2537 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 14:09:24.223799 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223755    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/edd8cf01-23e4-4cca-bce4-52210d98b048-root\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.223799 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:09:24.223790    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-tls podName:edd8cf01-23e4-4cca-bce4-52210d98b048 nodeName:}" failed. No retries permitted until 2026-09-17 14:09:24.723767176 +0000 UTC m=+169.014881018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-tls") pod "node-exporter-bwhn7" (UID: "edd8cf01-23e4-4cca-bce4-52210d98b048") : secret "node-exporter-tls" not found
Sep 17 14:09:24.224145 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223806    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-wtmp\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.224145 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223827    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edd8cf01-23e4-4cca-bce4-52210d98b048-sys\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.224145 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223858    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edd8cf01-23e4-4cca-bce4-52210d98b048-sys\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.224145 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223806    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/edd8cf01-23e4-4cca-bce4-52210d98b048-root\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.224145 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.223992    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-textfile\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.224331 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.224280    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-accelerators-collector-config\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.224380 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.224365    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edd8cf01-23e4-4cca-bce4-52210d98b048-metrics-client-ca\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.226104 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.226086    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.244519 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.244498    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zr9f\" (UniqueName: \"kubernetes.io/projected/edd8cf01-23e4-4cca-bce4-52210d98b048-kube-api-access-9zr9f\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.727502 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.727469    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-tls\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.729657 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.729638    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edd8cf01-23e4-4cca-bce4-52210d98b048-node-exporter-tls\") pod \"node-exporter-bwhn7\" (UID: \"edd8cf01-23e4-4cca-bce4-52210d98b048\") " pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.927559 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:24.927510    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bwhn7"
Sep 17 14:09:24.940651 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:09:24.940624    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd8cf01_23e4_4cca_bce4_52210d98b048.slice/crio-54ae2450e938f0def96e696a4f030a031653354a428715c45c393b4a6eedf556 WatchSource:0}: Error finding container 54ae2450e938f0def96e696a4f030a031653354a428715c45c393b4a6eedf556: Status 404 returned error can't find the container with id 54ae2450e938f0def96e696a4f030a031653354a428715c45c393b4a6eedf556
Sep 17 14:09:25.886313 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:25.886278    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwhn7" event={"ID":"edd8cf01-23e4-4cca-bce4-52210d98b048","Type":"ContainerStarted","Data":"54ae2450e938f0def96e696a4f030a031653354a428715c45c393b4a6eedf556"}
Sep 17 14:09:26.891399 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:26.891363    2537 generic.go:356] "Generic (PLEG): container finished" podID="edd8cf01-23e4-4cca-bce4-52210d98b048" containerID="439b3c40d58c2d98259340d8f0d8f8bd9b6b23adc32c52e04b0fa7678e95521a" exitCode=0
Sep 17 14:09:26.891766 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:26.891425    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwhn7" event={"ID":"edd8cf01-23e4-4cca-bce4-52210d98b048","Type":"ContainerDied","Data":"439b3c40d58c2d98259340d8f0d8f8bd9b6b23adc32c52e04b0fa7678e95521a"}
Sep 17 14:09:27.896783 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:27.896742    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwhn7" event={"ID":"edd8cf01-23e4-4cca-bce4-52210d98b048","Type":"ContainerStarted","Data":"a7578032958cad2959864a7d47f718f8105b4efd107fb54b385007230e815317"}
Sep 17 14:09:27.896783 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:27.896790    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwhn7" event={"ID":"edd8cf01-23e4-4cca-bce4-52210d98b048","Type":"ContainerStarted","Data":"3940974a475b696c16293f8e5e85662c5ad093ecb67062ff49f51be263529f87"}
Sep 17 14:09:27.955426 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:27.955371    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bwhn7" podStartSLOduration=3.9475704929999997 podStartE2EDuration="4.955354262s" totalImagesPullingTime="948.322998ms" totalInitContainerRuntime="59.460771ms" isStatefulPod=true podCreationTimestamp="2026-09-17 14:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:27.954466977 +0000 UTC m=+172.245580842" watchObservedRunningTime="2026-09-17 14:09:27.955354262 +0000 UTC m=+172.246468146"
Sep 17 14:09:28.239534 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:28.239451    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:09:29.904205 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:29.904158    2537 generic.go:356] "Generic (PLEG): container finished" podID="db88b394-d715-41fc-80a1-bc2257b06186" containerID="fa86eb09802999b147f0a885ad8b6ec3c7cd5e3b96837c2d41ab9b75e137103f" exitCode=1
Sep 17 14:09:29.904518 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:29.904209    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj" event={"ID":"db88b394-d715-41fc-80a1-bc2257b06186","Type":"ContainerDied","Data":"fa86eb09802999b147f0a885ad8b6ec3c7cd5e3b96837c2d41ab9b75e137103f"}
Sep 17 14:09:29.904563 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:29.904520    2537 scope.go:117] "RemoveContainer" containerID="fa86eb09802999b147f0a885ad8b6ec3c7cd5e3b96837c2d41ab9b75e137103f"
Sep 17 14:09:30.523366 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:30.523334    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:09:30.872992 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:30.872964    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m4qxw"
Sep 17 14:09:30.909168 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:30.909137    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj" event={"ID":"db88b394-d715-41fc-80a1-bc2257b06186","Type":"ContainerStarted","Data":"c720f08e2c768685d761f7f0ba633961e2e2104cb8af1abc31d84a87c4d3e4ca"}
Sep 17 14:09:30.909613 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:30.909363    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:09:30.909923 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:30.909903    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77c97794cc-7nssj"
Sep 17 14:09:30.910820 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:30.910801    2537 generic.go:356] "Generic (PLEG): container finished" podID="f135e91d-d078-49d3-aca0-1cea8db0596f" containerID="4f3f55e63353f845aff13fafd482bb067fe5c38dcfc115c5de318161e97003c1" exitCode=255
Sep 17 14:09:30.910903 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:30.910855    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm" event={"ID":"f135e91d-d078-49d3-aca0-1cea8db0596f","Type":"ContainerDied","Data":"4f3f55e63353f845aff13fafd482bb067fe5c38dcfc115c5de318161e97003c1"}
Sep 17 14:09:30.911080 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:30.911069    2537 scope.go:117] "RemoveContainer" containerID="4f3f55e63353f845aff13fafd482bb067fe5c38dcfc115c5de318161e97003c1"
Sep 17 14:09:31.915753 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:31.915720    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866dcdf5f4-x8nrm" event={"ID":"f135e91d-d078-49d3-aca0-1cea8db0596f","Type":"ContainerStarted","Data":"a766cb5ee577b47f93e59ab0c7866a2f9ca50b12deba2bbeaa3de2047c77f753"}
Sep 17 14:09:37.655090 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:37.655054    2537 patch_prober.go:28] interesting pod/image-registry-648b8d4db-5pgz2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 14:09:37.655462 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:37.655107    2537 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2" podUID="1d4ece8e-8652-48fc-bd1e-77005ef9b11d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:09:38.162309 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:38.162280    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-648b8d4db-5pgz2"]
Sep 17 14:09:38.166403 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:38.166367    2537 patch_prober.go:28] interesting pod/image-registry-648b8d4db-5pgz2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 14:09:38.166535 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:38.166408    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2" podUID="1d4ece8e-8652-48fc-bd1e-77005ef9b11d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:09:48.166602 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:09:48.166572    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:10:03.180604 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.180562    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2" podUID="1d4ece8e-8652-48fc-bd1e-77005ef9b11d" containerName="registry" containerID="cri-o://9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98" gracePeriod=30
Sep 17 14:10:03.419572 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.419547    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:10:03.507583 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.507513    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-ca-trust-extracted\") pod \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") "
Sep 17 14:10:03.507583 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.507552    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-image-registry-private-configuration\") pod \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") "
Sep 17 14:10:03.507583 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.507572    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4kzn\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-kube-api-access-x4kzn\") pod \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") "
Sep 17 14:10:03.507835 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.507592    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-certificates\") pod \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") "
Sep 17 14:10:03.507835 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.507610    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-bound-sa-token\") pod \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") "
Sep 17 14:10:03.507835 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.507641    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-installation-pull-secrets\") pod \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") "
Sep 17 14:10:03.507835 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.507686    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-trusted-ca\") pod \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") "
Sep 17 14:10:03.507835 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.507716    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") pod \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\" (UID: \"1d4ece8e-8652-48fc-bd1e-77005ef9b11d\") "
Sep 17 14:10:03.508152 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.508117    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1d4ece8e-8652-48fc-bd1e-77005ef9b11d" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:10:03.508410 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.508380    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1d4ece8e-8652-48fc-bd1e-77005ef9b11d" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:10:03.510270 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.510236    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1d4ece8e-8652-48fc-bd1e-77005ef9b11d" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:10:03.510392 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.510332    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-kube-api-access-x4kzn" (OuterVolumeSpecName: "kube-api-access-x4kzn") pod "1d4ece8e-8652-48fc-bd1e-77005ef9b11d" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d"). InnerVolumeSpecName "kube-api-access-x4kzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:10:03.510392 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.510331    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1d4ece8e-8652-48fc-bd1e-77005ef9b11d" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:03.510503 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.510397    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1d4ece8e-8652-48fc-bd1e-77005ef9b11d" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:10:03.510503 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.510432    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1d4ece8e-8652-48fc-bd1e-77005ef9b11d" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:03.516270 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.516246    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1d4ece8e-8652-48fc-bd1e-77005ef9b11d" (UID: "1d4ece8e-8652-48fc-bd1e-77005ef9b11d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:10:03.608392 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.608353    2537 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-bound-sa-token\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:10:03.608392 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.608388    2537 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-installation-pull-secrets\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:10:03.608546 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.608402    2537 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-trusted-ca\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:10:03.608546 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.608415    2537 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-tls\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:10:03.608546 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.608427    2537 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-ca-trust-extracted\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:10:03.608546 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.608441    2537 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-image-registry-private-configuration\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:10:03.608546 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.608454    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4kzn\" (UniqueName: \"kubernetes.io/projected/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-kube-api-access-x4kzn\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:10:03.608546 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:03.608466    2537 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d4ece8e-8652-48fc-bd1e-77005ef9b11d-registry-certificates\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:10:04.017124 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:04.017089    2537 generic.go:356] "Generic (PLEG): container finished" podID="1d4ece8e-8652-48fc-bd1e-77005ef9b11d" containerID="9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98" exitCode=0
Sep 17 14:10:04.017296 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:04.017130    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2"
Sep 17 14:10:04.017296 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:04.017152    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2" event={"ID":"1d4ece8e-8652-48fc-bd1e-77005ef9b11d","Type":"ContainerDied","Data":"9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98"}
Sep 17 14:10:04.017296 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:04.017208    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648b8d4db-5pgz2" event={"ID":"1d4ece8e-8652-48fc-bd1e-77005ef9b11d","Type":"ContainerDied","Data":"54bd23ff731d5d229510d560f712089076582d6c64072c0fd6f98a4ddcfc1660"}
Sep 17 14:10:04.017296 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:04.017227    2537 scope.go:117] "RemoveContainer" containerID="9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98"
Sep 17 14:10:04.025394 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:04.025374    2537 scope.go:117] "RemoveContainer" containerID="9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98"
Sep 17 14:10:04.025659 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:10:04.025636    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98\": container with ID starting with 9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98 not found: ID does not exist" containerID="9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98"
Sep 17 14:10:04.025725 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:04.025671    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98"} err="failed to get container status \"9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98\": rpc error: code = NotFound desc = could not find container \"9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98\": container with ID starting with 9936a6d7482ed6e01a4eba27ebcf4722e8e41847fb1401f25d60b86654c9ad98 not found: ID does not exist"
Sep 17 14:10:04.038190 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:04.038155    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-648b8d4db-5pgz2"]
Sep 17 14:10:04.042903 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:04.042883    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-648b8d4db-5pgz2"]
Sep 17 14:10:04.243256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:04.243223    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d4ece8e-8652-48fc-bd1e-77005ef9b11d" path="/var/lib/kubelet/pods/1d4ece8e-8652-48fc-bd1e-77005ef9b11d/volumes"
Sep 17 14:10:48.026440 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:48.026382    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:10:48.028735 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:48.028700    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d8f52c-5035-471e-b6b0-c95e222670a8-metrics-certs\") pod \"network-metrics-daemon-kp9pm\" (UID: \"d2d8f52c-5035-471e-b6b0-c95e222670a8\") " pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:10:48.043713 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:48.043689    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ck6nq\""
Sep 17 14:10:48.051235 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:48.051216    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kp9pm"
Sep 17 14:10:48.332319 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:48.330589    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kp9pm"]
Sep 17 14:10:48.336946 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:10:48.336911    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d8f52c_5035_471e_b6b0_c95e222670a8.slice/crio-e1b948e0c2f8f8ff028c95b5d9344f77dbd738d08bf38721eadbf8b2fc9b77de WatchSource:0}: Error finding container e1b948e0c2f8f8ff028c95b5d9344f77dbd738d08bf38721eadbf8b2fc9b77de: Status 404 returned error can't find the container with id e1b948e0c2f8f8ff028c95b5d9344f77dbd738d08bf38721eadbf8b2fc9b77de
Sep 17 14:10:49.161439 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:49.161389    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kp9pm" event={"ID":"d2d8f52c-5035-471e-b6b0-c95e222670a8","Type":"ContainerStarted","Data":"e1b948e0c2f8f8ff028c95b5d9344f77dbd738d08bf38721eadbf8b2fc9b77de"}
Sep 17 14:10:50.166327 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:50.166291    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kp9pm" event={"ID":"d2d8f52c-5035-471e-b6b0-c95e222670a8","Type":"ContainerStarted","Data":"cb602ece6bb86de4bec43334adf14faa76d5fd1d54b18c53cd9a2580640bf969"}
Sep 17 14:10:50.166807 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:50.166351    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kp9pm" event={"ID":"d2d8f52c-5035-471e-b6b0-c95e222670a8","Type":"ContainerStarted","Data":"242c8a37542c68c66b5cbc19bc492d12b573bf506c67e9f9e75504f619c31387"}
Sep 17 14:10:50.184139 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:10:50.184094    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kp9pm" podStartSLOduration=253.231991492 podStartE2EDuration="4m14.184080211s" totalImagesPullingTime="952.088719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:10:50.18248972 +0000 UTC m=+254.473603605" watchObservedRunningTime="2026-09-17 14:10:50.184080211 +0000 UTC m=+254.475194072"
Sep 17 14:11:36.073871 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:11:36.073842    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:11:36.077452 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:11:36.077422    2537 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 14:11:36.080789 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:11:36.080770    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:14:12.163893 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.163810    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-29l6s"]
Sep 17 14:14:12.164393 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.164100    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4ece8e-8652-48fc-bd1e-77005ef9b11d" containerName="registry"
Sep 17 14:14:12.164393 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.164115    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4ece8e-8652-48fc-bd1e-77005ef9b11d" containerName="registry"
Sep 17 14:14:12.164393 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.164171    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4ece8e-8652-48fc-bd1e-77005ef9b11d" containerName="registry"
Sep 17 14:14:12.166949 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.166934    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-29l6s"
Sep 17 14:14:12.169751 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.169728    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 14:14:12.169913 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.169901    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 14:14:12.171122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.171105    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-27bnm\""
Sep 17 14:14:12.176087 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.176066    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-29l6s"]
Sep 17 14:14:12.267277 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.267254    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2ff2932-48d9-4819-aa98-a080799e6fdc-bound-sa-token\") pod \"cert-manager-55b4d954d5-29l6s\" (UID: \"b2ff2932-48d9-4819-aa98-a080799e6fdc\") " pod="cert-manager/cert-manager-55b4d954d5-29l6s"
Sep 17 14:14:12.267410 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.267288    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnhc\" (UniqueName: \"kubernetes.io/projected/b2ff2932-48d9-4819-aa98-a080799e6fdc-kube-api-access-jvnhc\") pod \"cert-manager-55b4d954d5-29l6s\" (UID: \"b2ff2932-48d9-4819-aa98-a080799e6fdc\") " pod="cert-manager/cert-manager-55b4d954d5-29l6s"
Sep 17 14:14:12.368229 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.368205    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2ff2932-48d9-4819-aa98-a080799e6fdc-bound-sa-token\") pod \"cert-manager-55b4d954d5-29l6s\" (UID: \"b2ff2932-48d9-4819-aa98-a080799e6fdc\") " pod="cert-manager/cert-manager-55b4d954d5-29l6s"
Sep 17 14:14:12.368358 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.368256    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnhc\" (UniqueName: \"kubernetes.io/projected/b2ff2932-48d9-4819-aa98-a080799e6fdc-kube-api-access-jvnhc\") pod \"cert-manager-55b4d954d5-29l6s\" (UID: \"b2ff2932-48d9-4819-aa98-a080799e6fdc\") " pod="cert-manager/cert-manager-55b4d954d5-29l6s"
Sep 17 14:14:12.378092 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.378068    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2ff2932-48d9-4819-aa98-a080799e6fdc-bound-sa-token\") pod \"cert-manager-55b4d954d5-29l6s\" (UID: \"b2ff2932-48d9-4819-aa98-a080799e6fdc\") " pod="cert-manager/cert-manager-55b4d954d5-29l6s"
Sep 17 14:14:12.378213 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.378072    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnhc\" (UniqueName: \"kubernetes.io/projected/b2ff2932-48d9-4819-aa98-a080799e6fdc-kube-api-access-jvnhc\") pod \"cert-manager-55b4d954d5-29l6s\" (UID: \"b2ff2932-48d9-4819-aa98-a080799e6fdc\") " pod="cert-manager/cert-manager-55b4d954d5-29l6s"
Sep 17 14:14:12.475320 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.475244    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-29l6s"
Sep 17 14:14:12.747616 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.747548    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-29l6s"]
Sep 17 14:14:12.750723 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:14:12.750699    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2ff2932_48d9_4819_aa98_a080799e6fdc.slice/crio-6dfa3321507010afc179df229b7b6966f97f965a767c433a712c24feddd4326e WatchSource:0}: Error finding container 6dfa3321507010afc179df229b7b6966f97f965a767c433a712c24feddd4326e: Status 404 returned error can't find the container with id 6dfa3321507010afc179df229b7b6966f97f965a767c433a712c24feddd4326e
Sep 17 14:14:12.753082 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.753067    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:14:12.812964 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:12.812932    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-29l6s" event={"ID":"b2ff2932-48d9-4819-aa98-a080799e6fdc","Type":"ContainerStarted","Data":"6dfa3321507010afc179df229b7b6966f97f965a767c433a712c24feddd4326e"}
Sep 17 14:14:16.833524 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:16.833484    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-29l6s" event={"ID":"b2ff2932-48d9-4819-aa98-a080799e6fdc","Type":"ContainerStarted","Data":"52b6a9d6457d3f4020ea64701681d05ae674ecced24276d19dee17a5b551d159"}
Sep 17 14:14:16.859098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:16.859052    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-29l6s" podStartSLOduration=1.6555442500000002 podStartE2EDuration="4.859035354s" totalImagesPullingTime="3.203491104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:14:16.858442276 +0000 UTC m=+461.149556137" watchObservedRunningTime="2026-09-17 14:14:16.859035354 +0000 UTC m=+461.150149217"
Sep 17 14:14:37.267247 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.267215    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"]
Sep 17 14:14:37.270511 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.270489    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.275075 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.275049    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 14:14:37.275280 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.275262    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 14:14:37.275380 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.275359    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fx4tc\""
Sep 17 14:14:37.276433 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.276404    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 14:14:37.278403 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.278383    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 14:14:37.292769 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.292743    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"]
Sep 17 14:14:37.329253 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.329220    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe7032d6-ba74-471f-90e2-b539862c19ec-tmp\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.329253 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.329253    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe7032d6-ba74-471f-90e2-b539862c19ec-manifests\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.329459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.329291    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe7032d6-ba74-471f-90e2-b539862c19ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.329459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.329315    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe7032d6-ba74-471f-90e2-b539862c19ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.329459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.329354    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b275p\" (UniqueName: \"kubernetes.io/projected/fe7032d6-ba74-471f-90e2-b539862c19ec-kube-api-access-b275p\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.429937 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.429903    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe7032d6-ba74-471f-90e2-b539862c19ec-tmp\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.429937 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.429935    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe7032d6-ba74-471f-90e2-b539862c19ec-manifests\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.430164 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.429958    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe7032d6-ba74-471f-90e2-b539862c19ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.430164 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.429982    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe7032d6-ba74-471f-90e2-b539862c19ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.430164 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.430000    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b275p\" (UniqueName: \"kubernetes.io/projected/fe7032d6-ba74-471f-90e2-b539862c19ec-kube-api-access-b275p\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.430408 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.430329    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe7032d6-ba74-471f-90e2-b539862c19ec-manifests\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.432280 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.432253    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe7032d6-ba74-471f-90e2-b539862c19ec-tmp\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.432465 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.432440    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe7032d6-ba74-471f-90e2-b539862c19ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.432558 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.432543    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe7032d6-ba74-471f-90e2-b539862c19ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.447758 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.447737    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b275p\" (UniqueName: \"kubernetes.io/projected/fe7032d6-ba74-471f-90e2-b539862c19ec-kube-api-access-b275p\") pod \"opendatahub-operator-controller-manager-569db487f9-q92qv\" (UID: \"fe7032d6-ba74-471f-90e2-b539862c19ec\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.580249 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.580163    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:37.859474 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.859450    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"]
Sep 17 14:14:37.862007 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:14:37.861972    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7032d6_ba74_471f_90e2_b539862c19ec.slice/crio-f12611076dca49a0ca74c421e0ad6f34ef7362554ac2eea27cea18eda21f6afd WatchSource:0}: Error finding container f12611076dca49a0ca74c421e0ad6f34ef7362554ac2eea27cea18eda21f6afd: Status 404 returned error can't find the container with id f12611076dca49a0ca74c421e0ad6f34ef7362554ac2eea27cea18eda21f6afd
Sep 17 14:14:37.902685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:37.902648    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv" event={"ID":"fe7032d6-ba74-471f-90e2-b539862c19ec","Type":"ContainerStarted","Data":"f12611076dca49a0ca74c421e0ad6f34ef7362554ac2eea27cea18eda21f6afd"}
Sep 17 14:14:40.918968 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:40.918937    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv" event={"ID":"fe7032d6-ba74-471f-90e2-b539862c19ec","Type":"ContainerStarted","Data":"0c64e8c44d02e3acd78b6aa2b0edef020a31fa66d39fae512e992df8f5c4242a"}
Sep 17 14:14:42.928815 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:42.928782    2537 generic.go:356] "Generic (PLEG): container finished" podID="fe7032d6-ba74-471f-90e2-b539862c19ec" containerID="0c64e8c44d02e3acd78b6aa2b0edef020a31fa66d39fae512e992df8f5c4242a" exitCode=0
Sep 17 14:14:42.929249 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:42.928834    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv" event={"ID":"fe7032d6-ba74-471f-90e2-b539862c19ec","Type":"ContainerDied","Data":"0c64e8c44d02e3acd78b6aa2b0edef020a31fa66d39fae512e992df8f5c4242a"}
Sep 17 14:14:43.934056 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:43.934023    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv" event={"ID":"fe7032d6-ba74-471f-90e2-b539862c19ec","Type":"ContainerStarted","Data":"3236845b2a16792ef438d94414eca6c7c338b2df71c3fcd2b42fb05a83b59868"}
Sep 17 14:14:43.934508 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:43.934233    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:54.940254 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:54.940223    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv"
Sep 17 14:14:54.973238 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:54.973170    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-q92qv" podStartSLOduration=13.606517915 podStartE2EDuration="17.973154172s" totalImagesPullingTime="3.318502879s" totalInitContainerRuntime="1.048133378s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:14:43.983373155 +0000 UTC m=+488.274487028" watchObservedRunningTime="2026-09-17 14:14:54.973154172 +0000 UTC m=+499.264268075"
Sep 17 14:14:57.403019 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.402980    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-549869464b-7pvhb"]
Sep 17 14:14:57.409972 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.409948    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:57.414574 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.414551    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 14:14:57.415208 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.415171    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 14:14:57.415522 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.415501    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 14:14:57.416062 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.416046    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 14:14:57.431686 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.431660    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-549869464b-7pvhb"]
Sep 17 14:14:57.580403 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.580370    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx9np\" (UniqueName: \"kubernetes.io/projected/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-kube-api-access-dx9np\") pod \"kube-auth-proxy-549869464b-7pvhb\" (UID: \"73a95568-9bbe-47dc-a8ea-de9b6a5eff04\") " pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:57.580557 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.580429    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-tls-certs\") pod \"kube-auth-proxy-549869464b-7pvhb\" (UID: \"73a95568-9bbe-47dc-a8ea-de9b6a5eff04\") " pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:57.580557 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.580488    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-tmp\") pod \"kube-auth-proxy-549869464b-7pvhb\" (UID: \"73a95568-9bbe-47dc-a8ea-de9b6a5eff04\") " pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:57.681775 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.681695    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-tls-certs\") pod \"kube-auth-proxy-549869464b-7pvhb\" (UID: \"73a95568-9bbe-47dc-a8ea-de9b6a5eff04\") " pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:57.681775 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.681739    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-tmp\") pod \"kube-auth-proxy-549869464b-7pvhb\" (UID: \"73a95568-9bbe-47dc-a8ea-de9b6a5eff04\") " pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:57.681775 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.681776    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9np\" (UniqueName: \"kubernetes.io/projected/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-kube-api-access-dx9np\") pod \"kube-auth-proxy-549869464b-7pvhb\" (UID: \"73a95568-9bbe-47dc-a8ea-de9b6a5eff04\") " pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:57.681958 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:14:57.681848    2537 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 17 14:14:57.681958 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:14:57.681928    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-tls-certs podName:73a95568-9bbe-47dc-a8ea-de9b6a5eff04 nodeName:}" failed. No retries permitted until 2026-09-17 14:14:58.181911802 +0000 UTC m=+502.473025654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-tls-certs") pod "kube-auth-proxy-549869464b-7pvhb" (UID: "73a95568-9bbe-47dc-a8ea-de9b6a5eff04") : secret "kube-auth-proxy-tls" not found
Sep 17 14:14:57.683936 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.683916    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-tmp\") pod \"kube-auth-proxy-549869464b-7pvhb\" (UID: \"73a95568-9bbe-47dc-a8ea-de9b6a5eff04\") " pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:57.699276 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:57.699253    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9np\" (UniqueName: \"kubernetes.io/projected/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-kube-api-access-dx9np\") pod \"kube-auth-proxy-549869464b-7pvhb\" (UID: \"73a95568-9bbe-47dc-a8ea-de9b6a5eff04\") " pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:58.186625 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:58.186582    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-tls-certs\") pod \"kube-auth-proxy-549869464b-7pvhb\" (UID: \"73a95568-9bbe-47dc-a8ea-de9b6a5eff04\") " pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:58.189335 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:58.189309    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a95568-9bbe-47dc-a8ea-de9b6a5eff04-tls-certs\") pod \"kube-auth-proxy-549869464b-7pvhb\" (UID: \"73a95568-9bbe-47dc-a8ea-de9b6a5eff04\") " pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:58.318691 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:58.318663    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb"
Sep 17 14:14:58.594202 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:58.594114    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-549869464b-7pvhb"]
Sep 17 14:14:58.597918 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:14:58.597893    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a95568_9bbe_47dc_a8ea_de9b6a5eff04.slice/crio-a89b2c08302e4e35b3fe576cbc4afdfa26cc00c47955747efef66375df3d5003 WatchSource:0}: Error finding container a89b2c08302e4e35b3fe576cbc4afdfa26cc00c47955747efef66375df3d5003: Status 404 returned error can't find the container with id a89b2c08302e4e35b3fe576cbc4afdfa26cc00c47955747efef66375df3d5003
Sep 17 14:14:58.985780 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:58.985745    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb" event={"ID":"73a95568-9bbe-47dc-a8ea-de9b6a5eff04","Type":"ContainerStarted","Data":"a89b2c08302e4e35b3fe576cbc4afdfa26cc00c47955747efef66375df3d5003"}
Sep 17 14:14:59.069684 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.069652    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"]
Sep 17 14:14:59.073616 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.073599    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.078850 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.078825    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 14:14:59.078976 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.078907    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 14:14:59.078976 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.078939    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 14:14:59.079069 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.078984    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 14:14:59.079371 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.079356    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d7f98\""
Sep 17 14:14:59.149140 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.149112    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"]
Sep 17 14:14:59.195079 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.195054    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.195213 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.195085    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.195213 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.195113    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.195213 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.195132    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.195213 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.195199    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.195346 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.195224    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-462g4\" (UniqueName: \"kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-kube-api-access-462g4\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.195346 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.195244    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.296613 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.296527    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.296613 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.296579    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.296613 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.296611    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.296867 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.296639    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.296867 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.296683    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.296867 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.296715    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-462g4\" (UniqueName: \"kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-kube-api-access-462g4\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.296867 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.296745    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.297633 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.297584    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.299892 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.299653    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.299892 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.299719    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.300518 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.300494    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.300518 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.300511    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.315776 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.315756    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-462g4\" (UniqueName: \"kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-kube-api-access-462g4\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.315877 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.315839    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-68p88\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.383062 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.382890    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:14:59.732920 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.732870    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"]
Sep 17 14:14:59.736892 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:14:59.736851    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af1a61c_66eb_4eee_aa8c_65fdf9e9df71.slice/crio-c763b58132ea6a6b4c37570671e5c2f25b149abd31a3589b1155b0cd35b6eeab WatchSource:0}: Error finding container c763b58132ea6a6b4c37570671e5c2f25b149abd31a3589b1155b0cd35b6eeab: Status 404 returned error can't find the container with id c763b58132ea6a6b4c37570671e5c2f25b149abd31a3589b1155b0cd35b6eeab
Sep 17 14:14:59.992214 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:14:59.992106    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88" event={"ID":"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71","Type":"ContainerStarted","Data":"c763b58132ea6a6b4c37570671e5c2f25b149abd31a3589b1155b0cd35b6eeab"}
Sep 17 14:15:02.002360 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:02.002296    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb" event={"ID":"73a95568-9bbe-47dc-a8ea-de9b6a5eff04","Type":"ContainerStarted","Data":"98058e05e1c4801fc1e555d92b2297686023c45d5ebe0645a12e0a055c474f60"}
Sep 17 14:15:02.021356 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:02.021291    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-549869464b-7pvhb" podStartSLOduration=2.281135123 podStartE2EDuration="5.021275208s" totalImagesPullingTime="2.740140085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:02.020017185 +0000 UTC m=+506.311131058" watchObservedRunningTime="2026-09-17 14:15:02.021275208 +0000 UTC m=+506.312389069"
Sep 17 14:15:03.063568 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:03.063529    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 14:15:03.063820 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:03.063597    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 14:15:04.013610 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:04.013575    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88" event={"ID":"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71","Type":"ContainerStarted","Data":"4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793"}
Sep 17 14:15:04.013802 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:04.013772    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:15:04.015115 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:04.015091    2537 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-68p88 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:15:04.015243 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:04.015138    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88" podUID="6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:15:04.045753 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:04.045701    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88" podStartSLOduration=1.721747755 podStartE2EDuration="5.045686586s" totalImagesPullingTime="3.323938831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:04.043364966 +0000 UTC m=+508.334478826" watchObservedRunningTime="2026-09-17 14:15:04.045686586 +0000 UTC m=+508.336800448"
Sep 17 14:15:05.017869 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:05.017837    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:15:07.949568 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:07.949533    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hxhh6"]
Sep 17 14:15:07.952864 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:07.952842    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6"
Sep 17 14:15:07.955811 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:07.955790    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 17 14:15:07.955944 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:07.955877    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4ghrt\""
Sep 17 14:15:07.962383 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:07.962356    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hxhh6"]
Sep 17 14:15:08.062702 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.062667    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4qwx\" (UniqueName: \"kubernetes.io/projected/c47cc71d-8718-4874-a0ed-8f1fb3e53be0-kube-api-access-d4qwx\") pod \"odh-model-controller-7998c67dc4-hxhh6\" (UID: \"c47cc71d-8718-4874-a0ed-8f1fb3e53be0\") " pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6"
Sep 17 14:15:08.062857 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.062712    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c47cc71d-8718-4874-a0ed-8f1fb3e53be0-cert\") pod \"odh-model-controller-7998c67dc4-hxhh6\" (UID: \"c47cc71d-8718-4874-a0ed-8f1fb3e53be0\") " pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6"
Sep 17 14:15:08.164056 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.164017    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4qwx\" (UniqueName: \"kubernetes.io/projected/c47cc71d-8718-4874-a0ed-8f1fb3e53be0-kube-api-access-d4qwx\") pod \"odh-model-controller-7998c67dc4-hxhh6\" (UID: \"c47cc71d-8718-4874-a0ed-8f1fb3e53be0\") " pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6"
Sep 17 14:15:08.164056 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.164062    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c47cc71d-8718-4874-a0ed-8f1fb3e53be0-cert\") pod \"odh-model-controller-7998c67dc4-hxhh6\" (UID: \"c47cc71d-8718-4874-a0ed-8f1fb3e53be0\") " pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6"
Sep 17 14:15:08.164341 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:15:08.164200    2537 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 14:15:08.164341 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:15:08.164253    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c47cc71d-8718-4874-a0ed-8f1fb3e53be0-cert podName:c47cc71d-8718-4874-a0ed-8f1fb3e53be0 nodeName:}" failed. No retries permitted until 2026-09-17 14:15:08.664238316 +0000 UTC m=+512.955352156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c47cc71d-8718-4874-a0ed-8f1fb3e53be0-cert") pod "odh-model-controller-7998c67dc4-hxhh6" (UID: "c47cc71d-8718-4874-a0ed-8f1fb3e53be0") : secret "odh-model-controller-webhook-cert" not found
Sep 17 14:15:08.175616 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.175590    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4qwx\" (UniqueName: \"kubernetes.io/projected/c47cc71d-8718-4874-a0ed-8f1fb3e53be0-kube-api-access-d4qwx\") pod \"odh-model-controller-7998c67dc4-hxhh6\" (UID: \"c47cc71d-8718-4874-a0ed-8f1fb3e53be0\") " pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6"
Sep 17 14:15:08.614917 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.614885    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ccvnw"]
Sep 17 14:15:08.617783 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.617768    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw"
Sep 17 14:15:08.621170 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.621142    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 17 14:15:08.621170 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.621155    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-gz6s8\""
Sep 17 14:15:08.630266 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.630239    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ccvnw"]
Sep 17 14:15:08.668021 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.667982    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c47cc71d-8718-4874-a0ed-8f1fb3e53be0-cert\") pod \"odh-model-controller-7998c67dc4-hxhh6\" (UID: \"c47cc71d-8718-4874-a0ed-8f1fb3e53be0\") " pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6"
Sep 17 14:15:08.670466 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.670436    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c47cc71d-8718-4874-a0ed-8f1fb3e53be0-cert\") pod \"odh-model-controller-7998c67dc4-hxhh6\" (UID: \"c47cc71d-8718-4874-a0ed-8f1fb3e53be0\") " pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6"
Sep 17 14:15:08.768985 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.768938    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98afef09-5ea8-49e4-b0fb-b11305bddd31-tls-certs\") pod \"model-serving-api-54d89bd5fc-ccvnw\" (UID: \"98afef09-5ea8-49e4-b0fb-b11305bddd31\") " pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw"
Sep 17 14:15:08.768985 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.768988    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l797t\" (UniqueName: \"kubernetes.io/projected/98afef09-5ea8-49e4-b0fb-b11305bddd31-kube-api-access-l797t\") pod \"model-serving-api-54d89bd5fc-ccvnw\" (UID: \"98afef09-5ea8-49e4-b0fb-b11305bddd31\") " pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw"
Sep 17 14:15:08.863960 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.863907    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6"
Sep 17 14:15:08.870143 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.870118    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98afef09-5ea8-49e4-b0fb-b11305bddd31-tls-certs\") pod \"model-serving-api-54d89bd5fc-ccvnw\" (UID: \"98afef09-5ea8-49e4-b0fb-b11305bddd31\") " pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw"
Sep 17 14:15:08.870231 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.870156    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l797t\" (UniqueName: \"kubernetes.io/projected/98afef09-5ea8-49e4-b0fb-b11305bddd31-kube-api-access-l797t\") pod \"model-serving-api-54d89bd5fc-ccvnw\" (UID: \"98afef09-5ea8-49e4-b0fb-b11305bddd31\") " pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw"
Sep 17 14:15:08.870336 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:15:08.870311    2537 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 14:15:08.870424 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:15:08.870411    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98afef09-5ea8-49e4-b0fb-b11305bddd31-tls-certs podName:98afef09-5ea8-49e4-b0fb-b11305bddd31 nodeName:}" failed. No retries permitted until 2026-09-17 14:15:09.370388865 +0000 UTC m=+513.661502707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/98afef09-5ea8-49e4-b0fb-b11305bddd31-tls-certs") pod "model-serving-api-54d89bd5fc-ccvnw" (UID: "98afef09-5ea8-49e4-b0fb-b11305bddd31") : secret "model-serving-api-tls" not found
Sep 17 14:15:08.881938 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:08.881913    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l797t\" (UniqueName: \"kubernetes.io/projected/98afef09-5ea8-49e4-b0fb-b11305bddd31-kube-api-access-l797t\") pod \"model-serving-api-54d89bd5fc-ccvnw\" (UID: \"98afef09-5ea8-49e4-b0fb-b11305bddd31\") " pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw"
Sep 17 14:15:09.139265 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:09.139237    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hxhh6"]
Sep 17 14:15:09.141803 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:15:09.141775    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47cc71d_8718_4874_a0ed_8f1fb3e53be0.slice/crio-1e236537a4e42c75e0c6df1b909d9407dc6749f286284487202963c7bf7ee4a9 WatchSource:0}: Error finding container 1e236537a4e42c75e0c6df1b909d9407dc6749f286284487202963c7bf7ee4a9: Status 404 returned error can't find the container with id 1e236537a4e42c75e0c6df1b909d9407dc6749f286284487202963c7bf7ee4a9
Sep 17 14:15:09.373767 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:09.373728    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98afef09-5ea8-49e4-b0fb-b11305bddd31-tls-certs\") pod \"model-serving-api-54d89bd5fc-ccvnw\" (UID: \"98afef09-5ea8-49e4-b0fb-b11305bddd31\") " pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw"
Sep 17 14:15:09.376205 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:09.376151    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98afef09-5ea8-49e4-b0fb-b11305bddd31-tls-certs\") pod \"model-serving-api-54d89bd5fc-ccvnw\" (UID: \"98afef09-5ea8-49e4-b0fb-b11305bddd31\") " pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw"
Sep 17 14:15:09.527693 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:09.527600    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw"
Sep 17 14:15:09.803810 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:09.803728    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ccvnw"]
Sep 17 14:15:09.806745 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:15:09.806714    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98afef09_5ea8_49e4_b0fb_b11305bddd31.slice/crio-bf390da8c8bfc5084e972b50d1abdb670d0ee94e48cb06c7ac110f7bb2405c97 WatchSource:0}: Error finding container bf390da8c8bfc5084e972b50d1abdb670d0ee94e48cb06c7ac110f7bb2405c97: Status 404 returned error can't find the container with id bf390da8c8bfc5084e972b50d1abdb670d0ee94e48cb06c7ac110f7bb2405c97
Sep 17 14:15:10.038583 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:10.038542    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw" event={"ID":"98afef09-5ea8-49e4-b0fb-b11305bddd31","Type":"ContainerStarted","Data":"bf390da8c8bfc5084e972b50d1abdb670d0ee94e48cb06c7ac110f7bb2405c97"}
Sep 17 14:15:10.040274 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:10.040201    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6" event={"ID":"c47cc71d-8718-4874-a0ed-8f1fb3e53be0","Type":"ContainerStarted","Data":"1e236537a4e42c75e0c6df1b909d9407dc6749f286284487202963c7bf7ee4a9"}
Sep 17 14:15:13.056914 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:13.056877    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw" event={"ID":"98afef09-5ea8-49e4-b0fb-b11305bddd31","Type":"ContainerStarted","Data":"b6b39a9c61ad2e20542299cdf1fd5b33b71b5e9bd1a7141b30f6f0ceeeed34c4"}
Sep 17 14:15:13.057410 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:13.057001    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw"
Sep 17 14:15:13.058641 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:13.058616    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6" event={"ID":"c47cc71d-8718-4874-a0ed-8f1fb3e53be0","Type":"ContainerStarted","Data":"e272391a3a994a84be3e9cd66ed0ba0f0fa858d589513025cf42356466ce9093"}
Sep 17 14:15:13.058803 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:13.058791    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6"
Sep 17 14:15:13.076245 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:13.076188    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw" podStartSLOduration=2.35245187 podStartE2EDuration="5.076153862s" totalImagesPullingTime="2.723701992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:13.074481107 +0000 UTC m=+517.365594968" watchObservedRunningTime="2026-09-17 14:15:13.076153862 +0000 UTC m=+517.367267715"
Sep 17 14:15:13.097663 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:13.097617    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6" podStartSLOduration=2.705841884 podStartE2EDuration="6.097602956s" totalImagesPullingTime="3.391761072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:13.096328129 +0000 UTC m=+517.387441993" watchObservedRunningTime="2026-09-17 14:15:13.097602956 +0000 UTC m=+517.388716854"
Sep 17 14:15:14.826660 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:14.826629    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"]
Sep 17 14:15:14.829862 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:14.829842    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:14.848658 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:14.848630    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"]
Sep 17 14:15:14.919684 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:14.919653    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:14.919684 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:14.919690    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:14.919882 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:14.919711    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:14.919882 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:14.919731    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:14.919882 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:14.919776    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:14.919882 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:14.919806    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:14.919882 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:14.919863    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqchb\" (UniqueName: \"kubernetes.io/projected/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-kube-api-access-qqchb\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.020281 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.020236    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.020281 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.020279    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.020519 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.020296    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.020519 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.020320    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqchb\" (UniqueName: \"kubernetes.io/projected/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-kube-api-access-qqchb\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.020519 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.020361    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.020519 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.020383    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.020519 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.020400    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.021075 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.021051    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.022610 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.022589    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.022759 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.022728    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.023043 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.023023    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.023043 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.023037    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.030121 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.030101    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqchb\" (UniqueName: \"kubernetes.io/projected/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-kube-api-access-qqchb\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.032418 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.032397    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc7a14-41f9-4205-89c6-3a3a4f61cbe5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2fmw5\" (UID: \"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.137814 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.137781    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:15.470165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.470094    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"]
Sep 17 14:15:15.473312 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:15:15.473279    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bc7a14_41f9_4205_89c6_3a3a4f61cbe5.slice/crio-794674e5266c716f3325e9e7c5827d462a4e6a435f02bf5bb8c9fee5670b3fd4 WatchSource:0}: Error finding container 794674e5266c716f3325e9e7c5827d462a4e6a435f02bf5bb8c9fee5670b3fd4: Status 404 returned error can't find the container with id 794674e5266c716f3325e9e7c5827d462a4e6a435f02bf5bb8c9fee5670b3fd4
Sep 17 14:15:15.475422 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.475391    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 14:15:15.475487 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:15.475455    2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 14:15:16.072369 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:16.072326    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5" event={"ID":"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5","Type":"ContainerStarted","Data":"a1c257999e3d65e95f3cb14203c297ffcb36c49e6bc98f306fd1e22b8ab9dc35"}
Sep 17 14:15:16.072807 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:16.072385    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5" event={"ID":"76bc7a14-41f9-4205-89c6-3a3a4f61cbe5","Type":"ContainerStarted","Data":"794674e5266c716f3325e9e7c5827d462a4e6a435f02bf5bb8c9fee5670b3fd4"}
Sep 17 14:15:16.072807 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:16.072488    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:16.074162 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:16.074139    2537 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-2fmw5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:15:16.074287 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:16.074217    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5" podUID="76bc7a14-41f9-4205-89c6-3a3a4f61cbe5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:15:17.077265 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.077235    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5"
Sep 17 14:15:17.295908 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.295846    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2fmw5" podStartSLOduration=3.295828236 podStartE2EDuration="3.295828236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:16.166337791 +0000 UTC m=+520.457451653" watchObservedRunningTime="2026-09-17 14:15:17.295828236 +0000 UTC m=+521.586942093"
Sep 17 14:15:17.584686 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.584648    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"]
Sep 17 14:15:17.584907 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.584861    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88" podUID="6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" containerName="discovery" containerID="cri-o://4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793" gracePeriod=30
Sep 17 14:15:17.828235 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.828209    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:15:17.840802 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.840729    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-token\") pod \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") "
Sep 17 14:15:17.840802 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.840774    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-ca-configmap\") pod \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") "
Sep 17 14:15:17.840802 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.840797    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-dns-cert\") pod \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") "
Sep 17 14:15:17.840999 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.840816    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-local-certs\") pod \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") "
Sep 17 14:15:17.840999 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.840907    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-462g4\" (UniqueName: \"kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-kube-api-access-462g4\") pod \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") "
Sep 17 14:15:17.840999 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.840949    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-cacerts\") pod \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") "
Sep 17 14:15:17.840999 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.840980    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-kubeconfig\") pod \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\" (UID: \"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71\") "
Sep 17 14:15:17.841286 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.841258    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" (UID: "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:15:17.843333 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.843294    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-token" (OuterVolumeSpecName: "istio-token") pod "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" (UID: "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:15:17.843333 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.843296    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-cacerts" (OuterVolumeSpecName: "cacerts") pod "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" (UID: "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:15:17.843506 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.843440    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-kube-api-access-462g4" (OuterVolumeSpecName: "kube-api-access-462g4") pod "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" (UID: "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71"). InnerVolumeSpecName "kube-api-access-462g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:15:17.843506 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.843488    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" (UID: "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:15:17.843596 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.843504    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-local-certs" (OuterVolumeSpecName: "local-certs") pod "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" (UID: "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:15:17.843596 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.843523    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" (UID: "6af1a61c-66eb-4eee-aa8c-65fdf9e9df71"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:15:17.941803 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.941765    2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-token\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:15:17.941803 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.941799    2537 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-ca-configmap\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:15:17.941803 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.941810    2537 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-csr-dns-cert\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:15:17.942033 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.941818    2537 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-local-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:15:17.942033 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.941827    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-462g4\" (UniqueName: \"kubernetes.io/projected/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-kube-api-access-462g4\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:15:17.942033 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.941835    2537 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-cacerts\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:15:17.942033 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:17.941846    2537 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71-istio-kubeconfig\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:15:18.081667 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:18.081634    2537 generic.go:356] "Generic (PLEG): container finished" podID="6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" containerID="4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793" exitCode=0
Sep 17 14:15:18.082054 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:18.081684    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"
Sep 17 14:15:18.082054 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:18.081723    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88" event={"ID":"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71","Type":"ContainerDied","Data":"4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793"}
Sep 17 14:15:18.082054 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:18.081774    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88" event={"ID":"6af1a61c-66eb-4eee-aa8c-65fdf9e9df71","Type":"ContainerDied","Data":"c763b58132ea6a6b4c37570671e5c2f25b149abd31a3589b1155b0cd35b6eeab"}
Sep 17 14:15:18.082054 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:18.081794    2537 scope.go:117] "RemoveContainer" containerID="4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793"
Sep 17 14:15:18.090293 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:18.090276    2537 scope.go:117] "RemoveContainer" containerID="4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793"
Sep 17 14:15:18.090555 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:15:18.090539    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793\": container with ID starting with 4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793 not found: ID does not exist" containerID="4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793"
Sep 17 14:15:18.090615 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:18.090561    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793"} err="failed to get container status \"4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793\": rpc error: code = NotFound desc = could not find container \"4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793\": container with ID starting with 4ee54f0b72ec052c08a3a30820ac390abf55cd041adc8410e83df91d15d44793 not found: ID does not exist"
Sep 17 14:15:18.123355 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:18.123269    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"]
Sep 17 14:15:18.127160 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:18.127126    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-68p88"]
Sep 17 14:15:18.243938 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:18.243903    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" path="/var/lib/kubelet/pods/6af1a61c-66eb-4eee-aa8c-65fdf9e9df71/volumes"
Sep 17 14:15:19.195686 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.195653    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"]
Sep 17 14:15:19.196103 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.195918    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" containerName="discovery"
Sep 17 14:15:19.196103 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.195931    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" containerName="discovery"
Sep 17 14:15:19.196103 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.195979    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af1a61c-66eb-4eee-aa8c-65fdf9e9df71" containerName="discovery"
Sep 17 14:15:19.200075 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.200059    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"
Sep 17 14:15:19.203680 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.203649    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pglgw\""
Sep 17 14:15:19.203680 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.203655    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 17 14:15:19.223496 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.223466    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"]
Sep 17 14:15:19.250375 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.250344    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxgv7\" (UniqueName: \"kubernetes.io/projected/3164cd21-60ad-4914-ac6f-769b3489289e-kube-api-access-wxgv7\") pod \"kserve-controller-manager-d6f4b65b8-j4c8c\" (UID: \"3164cd21-60ad-4914-ac6f-769b3489289e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"
Sep 17 14:15:19.250545 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.250404    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3164cd21-60ad-4914-ac6f-769b3489289e-cert\") pod \"kserve-controller-manager-d6f4b65b8-j4c8c\" (UID: \"3164cd21-60ad-4914-ac6f-769b3489289e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"
Sep 17 14:15:19.351159 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.351127    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3164cd21-60ad-4914-ac6f-769b3489289e-cert\") pod \"kserve-controller-manager-d6f4b65b8-j4c8c\" (UID: \"3164cd21-60ad-4914-ac6f-769b3489289e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"
Sep 17 14:15:19.351353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.351215    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxgv7\" (UniqueName: \"kubernetes.io/projected/3164cd21-60ad-4914-ac6f-769b3489289e-kube-api-access-wxgv7\") pod \"kserve-controller-manager-d6f4b65b8-j4c8c\" (UID: \"3164cd21-60ad-4914-ac6f-769b3489289e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"
Sep 17 14:15:19.351353 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:15:19.351306    2537 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 14:15:19.351479 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:15:19.351383    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3164cd21-60ad-4914-ac6f-769b3489289e-cert podName:3164cd21-60ad-4914-ac6f-769b3489289e nodeName:}" failed. No retries permitted until 2026-09-17 14:15:19.851362911 +0000 UTC m=+524.142476769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3164cd21-60ad-4914-ac6f-769b3489289e-cert") pod "kserve-controller-manager-d6f4b65b8-j4c8c" (UID: "3164cd21-60ad-4914-ac6f-769b3489289e") : secret "kserve-webhook-server-cert" not found
Sep 17 14:15:19.369061 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.369033    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxgv7\" (UniqueName: \"kubernetes.io/projected/3164cd21-60ad-4914-ac6f-769b3489289e-kube-api-access-wxgv7\") pod \"kserve-controller-manager-d6f4b65b8-j4c8c\" (UID: \"3164cd21-60ad-4914-ac6f-769b3489289e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"
Sep 17 14:15:19.854359 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.854314    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3164cd21-60ad-4914-ac6f-769b3489289e-cert\") pod \"kserve-controller-manager-d6f4b65b8-j4c8c\" (UID: \"3164cd21-60ad-4914-ac6f-769b3489289e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"
Sep 17 14:15:19.856670 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:19.856639    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3164cd21-60ad-4914-ac6f-769b3489289e-cert\") pod \"kserve-controller-manager-d6f4b65b8-j4c8c\" (UID: \"3164cd21-60ad-4914-ac6f-769b3489289e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"
Sep 17 14:15:20.109490 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:20.109406    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"
Sep 17 14:15:20.381670 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:20.381643    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"]
Sep 17 14:15:20.383348 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:15:20.383316    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3164cd21_60ad_4914_ac6f_769b3489289e.slice/crio-82bd6d447a596d21485b15297debe6e92d061477bbf8eb287b9a0f5060ae5ad4 WatchSource:0}: Error finding container 82bd6d447a596d21485b15297debe6e92d061477bbf8eb287b9a0f5060ae5ad4: Status 404 returned error can't find the container with id 82bd6d447a596d21485b15297debe6e92d061477bbf8eb287b9a0f5060ae5ad4
Sep 17 14:15:21.095752 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:21.095715    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c" event={"ID":"3164cd21-60ad-4914-ac6f-769b3489289e","Type":"ContainerStarted","Data":"82bd6d447a596d21485b15297debe6e92d061477bbf8eb287b9a0f5060ae5ad4"}
Sep 17 14:15:23.106793 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:23.106758    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c" event={"ID":"3164cd21-60ad-4914-ac6f-769b3489289e","Type":"ContainerStarted","Data":"7237f8a0170abd4a33a4d0daf27ef20403c2213346b45bd41ace2349b0013d29"}
Sep 17 14:15:23.107150 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:23.106908    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"
Sep 17 14:15:23.133857 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:23.133805    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c" podStartSLOduration=2.047176305 podStartE2EDuration="4.133789924s" totalImagesPullingTime="2.086613619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:23.132627117 +0000 UTC m=+527.423740978" watchObservedRunningTime="2026-09-17 14:15:23.133789924 +0000 UTC m=+527.424903816"
Sep 17 14:15:24.064364 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:24.064337    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-hxhh6"
Sep 17 14:15:24.066619 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:24.066596    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ccvnw"
Sep 17 14:15:54.117304 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:15:54.117270    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j4c8c"
Sep 17 14:16:31.370405 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:31.370372    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4hmzt"]
Sep 17 14:16:31.373369 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:31.373352    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-4hmzt"
Sep 17 14:16:31.389071 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:31.389052    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-q8v98\""
Sep 17 14:16:31.390231 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:31.390217    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 14:16:31.390302 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:31.390283    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 14:16:31.399591 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:31.399572    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv9hx\" (UniqueName: \"kubernetes.io/projected/aa5f0f44-e2ab-4245-aa4e-2dd45d732988-kube-api-access-pv9hx\") pod \"authorino-operator-76b478bd78-4hmzt\" (UID: \"aa5f0f44-e2ab-4245-aa4e-2dd45d732988\") " pod="kuadrant-system/authorino-operator-76b478bd78-4hmzt"
Sep 17 14:16:31.413856 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:31.413826    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4hmzt"]
Sep 17 14:16:31.500361 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:31.500320    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv9hx\" (UniqueName: \"kubernetes.io/projected/aa5f0f44-e2ab-4245-aa4e-2dd45d732988-kube-api-access-pv9hx\") pod \"authorino-operator-76b478bd78-4hmzt\" (UID: \"aa5f0f44-e2ab-4245-aa4e-2dd45d732988\") " pod="kuadrant-system/authorino-operator-76b478bd78-4hmzt"
Sep 17 14:16:31.524771 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:31.524738    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv9hx\" (UniqueName: \"kubernetes.io/projected/aa5f0f44-e2ab-4245-aa4e-2dd45d732988-kube-api-access-pv9hx\") pod \"authorino-operator-76b478bd78-4hmzt\" (UID: \"aa5f0f44-e2ab-4245-aa4e-2dd45d732988\") " pod="kuadrant-system/authorino-operator-76b478bd78-4hmzt"
Sep 17 14:16:31.682600 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:31.682521    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-4hmzt"
Sep 17 14:16:31.978891 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:31.975518    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4hmzt"]
Sep 17 14:16:31.982837 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:16:31.982808    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5f0f44_e2ab_4245_aa4e_2dd45d732988.slice/crio-c20df81a054447d68d485b9ce03fa4bbce6fc0a8fe644b589ddeee795b38f4ea WatchSource:0}: Error finding container c20df81a054447d68d485b9ce03fa4bbce6fc0a8fe644b589ddeee795b38f4ea: Status 404 returned error can't find the container with id c20df81a054447d68d485b9ce03fa4bbce6fc0a8fe644b589ddeee795b38f4ea
Sep 17 14:16:32.363791 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:32.363757    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-4hmzt" event={"ID":"aa5f0f44-e2ab-4245-aa4e-2dd45d732988","Type":"ContainerStarted","Data":"c20df81a054447d68d485b9ce03fa4bbce6fc0a8fe644b589ddeee795b38f4ea"}
Sep 17 14:16:33.218968 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:33.218925    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r"]
Sep 17 14:16:33.227608 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:33.227581    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r"
Sep 17 14:16:33.231306 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:33.231270    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6hrsw\""
Sep 17 14:16:33.231920 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:33.231901    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 17 14:16:33.246637 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:33.246605    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r"]
Sep 17 14:16:33.313075 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:33.313033    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmp6w\" (UniqueName: \"kubernetes.io/projected/6fe52c70-f73c-4698-bd86-2fa571da0689-kube-api-access-mmp6w\") pod \"dns-operator-controller-manager-7b84cd7c9c-wmq5r\" (UID: \"6fe52c70-f73c-4698-bd86-2fa571da0689\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r"
Sep 17 14:16:33.414074 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:33.414035    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmp6w\" (UniqueName: \"kubernetes.io/projected/6fe52c70-f73c-4698-bd86-2fa571da0689-kube-api-access-mmp6w\") pod \"dns-operator-controller-manager-7b84cd7c9c-wmq5r\" (UID: \"6fe52c70-f73c-4698-bd86-2fa571da0689\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r"
Sep 17 14:16:33.429980 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:33.429944    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmp6w\" (UniqueName: \"kubernetes.io/projected/6fe52c70-f73c-4698-bd86-2fa571da0689-kube-api-access-mmp6w\") pod \"dns-operator-controller-manager-7b84cd7c9c-wmq5r\" (UID: \"6fe52c70-f73c-4698-bd86-2fa571da0689\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r"
Sep 17 14:16:33.539166 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:33.539093    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r"
Sep 17 14:16:34.231653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:34.231564    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r"]
Sep 17 14:16:34.235419 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:16:34.235393    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe52c70_f73c_4698_bd86_2fa571da0689.slice/crio-9f89982a12eda0dc205e8c737912ca57895d0daffc5ad8555734055935bc903f WatchSource:0}: Error finding container 9f89982a12eda0dc205e8c737912ca57895d0daffc5ad8555734055935bc903f: Status 404 returned error can't find the container with id 9f89982a12eda0dc205e8c737912ca57895d0daffc5ad8555734055935bc903f
Sep 17 14:16:34.373408 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:34.373369    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r" event={"ID":"6fe52c70-f73c-4698-bd86-2fa571da0689","Type":"ContainerStarted","Data":"9f89982a12eda0dc205e8c737912ca57895d0daffc5ad8555734055935bc903f"}
Sep 17 14:16:34.375015 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:34.374989    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-4hmzt" event={"ID":"aa5f0f44-e2ab-4245-aa4e-2dd45d732988","Type":"ContainerStarted","Data":"60f4155f297788f21c51cde460a43055f4cfded063680d06a556df9559d2db33"}
Sep 17 14:16:34.375167 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:34.375119    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-4hmzt"
Sep 17 14:16:34.404836 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:34.404783    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-4hmzt" podStartSLOduration=1.3677678549999999 podStartE2EDuration="3.404764452s" totalImagesPullingTime="2.036996597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:34.403913215 +0000 UTC m=+598.695027077" watchObservedRunningTime="2026-09-17 14:16:34.404764452 +0000 UTC m=+598.695878314"
Sep 17 14:16:36.105934 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:36.105909    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:16:36.109286 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:36.109263    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:16:36.384467 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:36.384381    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r" event={"ID":"6fe52c70-f73c-4698-bd86-2fa571da0689","Type":"ContainerStarted","Data":"99597b553b48d317a198b93144097b830979d734f9eacf3a074813d72a9e4b32"}
Sep 17 14:16:36.384614 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:36.384504    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r"
Sep 17 14:16:36.409878 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:36.409828    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r" podStartSLOduration=1.903450072 podStartE2EDuration="3.409812989s" totalImagesPullingTime="1.506362917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:36.407678485 +0000 UTC m=+600.698792346" watchObservedRunningTime="2026-09-17 14:16:36.409812989 +0000 UTC m=+600.700926849"
Sep 17 14:16:45.382129 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:45.382090    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-4hmzt"
Sep 17 14:16:47.391047 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:47.391016    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wmq5r"
Sep 17 14:16:51.513753 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.513718    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"]
Sep 17 14:16:51.520240 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.520221    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:51.529019 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.529000    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xk98g\""
Sep 17 14:16:51.530212 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.530168    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0836c401-7c66-4b1a-93d9-bc18ead0bba7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4cslc\" (UID: \"0836c401-7c66-4b1a-93d9-bc18ead0bba7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:51.530310 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.530222    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb86k\" (UniqueName: \"kubernetes.io/projected/0836c401-7c66-4b1a-93d9-bc18ead0bba7-kube-api-access-gb86k\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4cslc\" (UID: \"0836c401-7c66-4b1a-93d9-bc18ead0bba7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:51.542198 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.542159    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"]
Sep 17 14:16:51.631576 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.631539    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0836c401-7c66-4b1a-93d9-bc18ead0bba7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4cslc\" (UID: \"0836c401-7c66-4b1a-93d9-bc18ead0bba7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:51.631755 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.631584    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb86k\" (UniqueName: \"kubernetes.io/projected/0836c401-7c66-4b1a-93d9-bc18ead0bba7-kube-api-access-gb86k\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4cslc\" (UID: \"0836c401-7c66-4b1a-93d9-bc18ead0bba7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:51.631924 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.631903    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0836c401-7c66-4b1a-93d9-bc18ead0bba7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4cslc\" (UID: \"0836c401-7c66-4b1a-93d9-bc18ead0bba7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:51.670715 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.670681    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"]
Sep 17 14:16:51.671004 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:16:51.670973    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gb86k], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc" podUID="0836c401-7c66-4b1a-93d9-bc18ead0bba7"
Sep 17 14:16:51.671665 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.671641    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb86k\" (UniqueName: \"kubernetes.io/projected/0836c401-7c66-4b1a-93d9-bc18ead0bba7-kube-api-access-gb86k\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4cslc\" (UID: \"0836c401-7c66-4b1a-93d9-bc18ead0bba7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:51.672379 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.672354    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"]
Sep 17 14:16:51.796095 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.796013    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"]
Sep 17 14:16:51.799965 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.799945    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"
Sep 17 14:16:51.833873 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.833837    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbbtj\" (UniqueName: \"kubernetes.io/projected/31d2366f-b00f-427d-9003-4fe8a8cb523a-kube-api-access-rbbtj\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xtnds\" (UID: \"31d2366f-b00f-427d-9003-4fe8a8cb523a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"
Sep 17 14:16:51.834041 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.833916    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31d2366f-b00f-427d-9003-4fe8a8cb523a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xtnds\" (UID: \"31d2366f-b00f-427d-9003-4fe8a8cb523a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"
Sep 17 14:16:51.851370 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.851338    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"]
Sep 17 14:16:51.935057 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.935023    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31d2366f-b00f-427d-9003-4fe8a8cb523a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xtnds\" (UID: \"31d2366f-b00f-427d-9003-4fe8a8cb523a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"
Sep 17 14:16:51.935270 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.935079    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbbtj\" (UniqueName: \"kubernetes.io/projected/31d2366f-b00f-427d-9003-4fe8a8cb523a-kube-api-access-rbbtj\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xtnds\" (UID: \"31d2366f-b00f-427d-9003-4fe8a8cb523a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"
Sep 17 14:16:51.935469 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.935444    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31d2366f-b00f-427d-9003-4fe8a8cb523a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xtnds\" (UID: \"31d2366f-b00f-427d-9003-4fe8a8cb523a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"
Sep 17 14:16:51.939438 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.939410    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"]
Sep 17 14:16:51.939723 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:16:51.939700    2537 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rbbtj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds" podUID="31d2366f-b00f-427d-9003-4fe8a8cb523a"
Sep 17 14:16:51.985815 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:51.985778    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbbtj\" (UniqueName: \"kubernetes.io/projected/31d2366f-b00f-427d-9003-4fe8a8cb523a-kube-api-access-rbbtj\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xtnds\" (UID: \"31d2366f-b00f-427d-9003-4fe8a8cb523a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"
Sep 17 14:16:52.448004 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.447975    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"
Sep 17 14:16:52.448169 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.447975    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:52.457106 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.457084    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"
Sep 17 14:16:52.459003 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.458987    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:52.501313 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:16:52.501281    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-4cslc\" is forbidden: User \"system:node:ip-10-0-133-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-98.ec2.internal' and this object" podUID="0836c401-7c66-4b1a-93d9-bc18ead0bba7" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:52.540043 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.540010    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0836c401-7c66-4b1a-93d9-bc18ead0bba7-extensions-socket-volume\") pod \"0836c401-7c66-4b1a-93d9-bc18ead0bba7\" (UID: \"0836c401-7c66-4b1a-93d9-bc18ead0bba7\") "
Sep 17 14:16:52.540444 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.540050    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb86k\" (UniqueName: \"kubernetes.io/projected/0836c401-7c66-4b1a-93d9-bc18ead0bba7-kube-api-access-gb86k\") pod \"0836c401-7c66-4b1a-93d9-bc18ead0bba7\" (UID: \"0836c401-7c66-4b1a-93d9-bc18ead0bba7\") "
Sep 17 14:16:52.540444 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.540081    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbbtj\" (UniqueName: \"kubernetes.io/projected/31d2366f-b00f-427d-9003-4fe8a8cb523a-kube-api-access-rbbtj\") pod \"31d2366f-b00f-427d-9003-4fe8a8cb523a\" (UID: \"31d2366f-b00f-427d-9003-4fe8a8cb523a\") "
Sep 17 14:16:52.540444 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.540146    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31d2366f-b00f-427d-9003-4fe8a8cb523a-extensions-socket-volume\") pod \"31d2366f-b00f-427d-9003-4fe8a8cb523a\" (UID: \"31d2366f-b00f-427d-9003-4fe8a8cb523a\") "
Sep 17 14:16:52.540444 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.540316    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0836c401-7c66-4b1a-93d9-bc18ead0bba7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0836c401-7c66-4b1a-93d9-bc18ead0bba7" (UID: "0836c401-7c66-4b1a-93d9-bc18ead0bba7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:16:52.540444 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.540393    2537 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0836c401-7c66-4b1a-93d9-bc18ead0bba7-extensions-socket-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:16:52.540668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.540492    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d2366f-b00f-427d-9003-4fe8a8cb523a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "31d2366f-b00f-427d-9003-4fe8a8cb523a" (UID: "31d2366f-b00f-427d-9003-4fe8a8cb523a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:16:52.542201 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.542163    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0836c401-7c66-4b1a-93d9-bc18ead0bba7-kube-api-access-gb86k" (OuterVolumeSpecName: "kube-api-access-gb86k") pod "0836c401-7c66-4b1a-93d9-bc18ead0bba7" (UID: "0836c401-7c66-4b1a-93d9-bc18ead0bba7"). InnerVolumeSpecName "kube-api-access-gb86k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:16:52.542274 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.542231    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d2366f-b00f-427d-9003-4fe8a8cb523a-kube-api-access-rbbtj" (OuterVolumeSpecName: "kube-api-access-rbbtj") pod "31d2366f-b00f-427d-9003-4fe8a8cb523a" (UID: "31d2366f-b00f-427d-9003-4fe8a8cb523a"). InnerVolumeSpecName "kube-api-access-rbbtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:16:52.641628 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.641586    2537 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31d2366f-b00f-427d-9003-4fe8a8cb523a-extensions-socket-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:16:52.641628 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.641619    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb86k\" (UniqueName: \"kubernetes.io/projected/0836c401-7c66-4b1a-93d9-bc18ead0bba7-kube-api-access-gb86k\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:16:52.641628 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:52.641631    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbbtj\" (UniqueName: \"kubernetes.io/projected/31d2366f-b00f-427d-9003-4fe8a8cb523a-kube-api-access-rbbtj\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:16:53.452836 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:53.452806    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:53.452994 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:53.452809    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"
Sep 17 14:16:53.463967 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:16:53.463940    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-4cslc\" is forbidden: User \"system:node:ip-10-0-133-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-98.ec2.internal' and this object" podUID="0836c401-7c66-4b1a-93d9-bc18ead0bba7" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:53.537910 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:53.537877    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"]
Sep 17 14:16:53.563628 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:53.563598    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xtnds"]
Sep 17 14:16:53.570995 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:16:53.570971    2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-4cslc\" is forbidden: User \"system:node:ip-10-0-133-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-98.ec2.internal' and this object" podUID="0836c401-7c66-4b1a-93d9-bc18ead0bba7" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4cslc"
Sep 17 14:16:54.245090 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:54.245056    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0836c401-7c66-4b1a-93d9-bc18ead0bba7" path="/var/lib/kubelet/pods/0836c401-7c66-4b1a-93d9-bc18ead0bba7/volumes"
Sep 17 14:16:54.245328 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:16:54.245315    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d2366f-b00f-427d-9003-4fe8a8cb523a" path="/var/lib/kubelet/pods/31d2366f-b00f-427d-9003-4fe8a8cb523a/volumes"
Sep 17 14:17:23.464830 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.464794    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-4hwhk"]
Sep 17 14:17:23.468053 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.468036    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:23.473542 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.473521    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g5n8f\""
Sep 17 14:17:23.473654 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.473577    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 14:17:23.492051 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.492017    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-4hwhk"]
Sep 17 14:17:23.552472 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.552444    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-4hwhk"]
Sep 17 14:17:23.585956 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.585922    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0ade81b-998b-45f3-a135-7dd1148ea74e-config-file\") pod \"limitador-limitador-8588749977-4hwhk\" (UID: \"c0ade81b-998b-45f3-a135-7dd1148ea74e\") " pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:23.586117 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.585988    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lftmc\" (UniqueName: \"kubernetes.io/projected/c0ade81b-998b-45f3-a135-7dd1148ea74e-kube-api-access-lftmc\") pod \"limitador-limitador-8588749977-4hwhk\" (UID: \"c0ade81b-998b-45f3-a135-7dd1148ea74e\") " pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:23.687078 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.687042    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lftmc\" (UniqueName: \"kubernetes.io/projected/c0ade81b-998b-45f3-a135-7dd1148ea74e-kube-api-access-lftmc\") pod \"limitador-limitador-8588749977-4hwhk\" (UID: \"c0ade81b-998b-45f3-a135-7dd1148ea74e\") " pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:23.687292 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.687088    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0ade81b-998b-45f3-a135-7dd1148ea74e-config-file\") pod \"limitador-limitador-8588749977-4hwhk\" (UID: \"c0ade81b-998b-45f3-a135-7dd1148ea74e\") " pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:23.687687 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.687667    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0ade81b-998b-45f3-a135-7dd1148ea74e-config-file\") pod \"limitador-limitador-8588749977-4hwhk\" (UID: \"c0ade81b-998b-45f3-a135-7dd1148ea74e\") " pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:23.699447 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.699425    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lftmc\" (UniqueName: \"kubernetes.io/projected/c0ade81b-998b-45f3-a135-7dd1148ea74e-kube-api-access-lftmc\") pod \"limitador-limitador-8588749977-4hwhk\" (UID: \"c0ade81b-998b-45f3-a135-7dd1148ea74e\") " pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:23.777530 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:23.777440    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:24.052227 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:17:24.052195    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ade81b_998b_45f3_a135_7dd1148ea74e.slice/crio-8bf13493e303ef3db6894ba2cf402480e3ddf7adfdaa2e39dbe7a52c0786aa31 WatchSource:0}: Error finding container 8bf13493e303ef3db6894ba2cf402480e3ddf7adfdaa2e39dbe7a52c0786aa31: Status 404 returned error can't find the container with id 8bf13493e303ef3db6894ba2cf402480e3ddf7adfdaa2e39dbe7a52c0786aa31
Sep 17 14:17:24.055961 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:24.055937    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-4hwhk"]
Sep 17 14:17:24.579928 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:24.579886    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk" event={"ID":"c0ade81b-998b-45f3-a135-7dd1148ea74e","Type":"ContainerStarted","Data":"8bf13493e303ef3db6894ba2cf402480e3ddf7adfdaa2e39dbe7a52c0786aa31"}
Sep 17 14:17:25.585895 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:25.585794    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk" event={"ID":"c0ade81b-998b-45f3-a135-7dd1148ea74e","Type":"ContainerStarted","Data":"b3248a5acb769d79ddd7738642c8b42538322edac1d21b79720fd6d33f6d0bb0"}
Sep 17 14:17:25.585895 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:25.585874    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:25.614134 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:25.614084    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk" podStartSLOduration=1.358020154 podStartE2EDuration="2.614067797s" totalImagesPullingTime="1.256047643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:17:25.611893716 +0000 UTC m=+649.903007590" watchObservedRunningTime="2026-09-17 14:17:25.614067797 +0000 UTC m=+649.905181662"
Sep 17 14:17:36.591324 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:36.591290    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:37.181950 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:37.181913    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-4hwhk"]
Sep 17 14:17:37.182135 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:37.182095    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk" podUID="c0ade81b-998b-45f3-a135-7dd1148ea74e" containerName="limitador" containerID="cri-o://b3248a5acb769d79ddd7738642c8b42538322edac1d21b79720fd6d33f6d0bb0" gracePeriod=30
Sep 17 14:17:37.635560 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:37.635521    2537 generic.go:356] "Generic (PLEG): container finished" podID="c0ade81b-998b-45f3-a135-7dd1148ea74e" containerID="b3248a5acb769d79ddd7738642c8b42538322edac1d21b79720fd6d33f6d0bb0" exitCode=0
Sep 17 14:17:37.635884 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:37.635592    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk" event={"ID":"c0ade81b-998b-45f3-a135-7dd1148ea74e","Type":"ContainerDied","Data":"b3248a5acb769d79ddd7738642c8b42538322edac1d21b79720fd6d33f6d0bb0"}
Sep 17 14:17:37.721480 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:37.721456    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:37.802462 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:37.802391    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0ade81b-998b-45f3-a135-7dd1148ea74e-config-file\") pod \"c0ade81b-998b-45f3-a135-7dd1148ea74e\" (UID: \"c0ade81b-998b-45f3-a135-7dd1148ea74e\") "
Sep 17 14:17:37.802462 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:37.802426    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lftmc\" (UniqueName: \"kubernetes.io/projected/c0ade81b-998b-45f3-a135-7dd1148ea74e-kube-api-access-lftmc\") pod \"c0ade81b-998b-45f3-a135-7dd1148ea74e\" (UID: \"c0ade81b-998b-45f3-a135-7dd1148ea74e\") "
Sep 17 14:17:37.802823 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:37.802795    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ade81b-998b-45f3-a135-7dd1148ea74e-config-file" (OuterVolumeSpecName: "config-file") pod "c0ade81b-998b-45f3-a135-7dd1148ea74e" (UID: "c0ade81b-998b-45f3-a135-7dd1148ea74e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:17:37.804570 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:37.804546    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ade81b-998b-45f3-a135-7dd1148ea74e-kube-api-access-lftmc" (OuterVolumeSpecName: "kube-api-access-lftmc") pod "c0ade81b-998b-45f3-a135-7dd1148ea74e" (UID: "c0ade81b-998b-45f3-a135-7dd1148ea74e"). InnerVolumeSpecName "kube-api-access-lftmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:17:37.903134 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:37.903104    2537 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0ade81b-998b-45f3-a135-7dd1148ea74e-config-file\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:17:37.903134 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:37.903130    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lftmc\" (UniqueName: \"kubernetes.io/projected/c0ade81b-998b-45f3-a135-7dd1148ea74e-kube-api-access-lftmc\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:17:38.641056 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:38.641024    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk" event={"ID":"c0ade81b-998b-45f3-a135-7dd1148ea74e","Type":"ContainerDied","Data":"8bf13493e303ef3db6894ba2cf402480e3ddf7adfdaa2e39dbe7a52c0786aa31"}
Sep 17 14:17:38.641478 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:38.641060    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-4hwhk"
Sep 17 14:17:38.641478 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:38.641085    2537 scope.go:117] "RemoveContainer" containerID="b3248a5acb769d79ddd7738642c8b42538322edac1d21b79720fd6d33f6d0bb0"
Sep 17 14:17:38.676002 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:38.675971    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-4hwhk"]
Sep 17 14:17:38.687489 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:38.687465    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-4hwhk"]
Sep 17 14:17:39.950150 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:39.950109    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"]
Sep 17 14:17:39.950660 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:39.950544    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ade81b-998b-45f3-a135-7dd1148ea74e" containerName="limitador"
Sep 17 14:17:39.950660 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:39.950563    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ade81b-998b-45f3-a135-7dd1148ea74e" containerName="limitador"
Sep 17 14:17:39.950660 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:39.950655    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ade81b-998b-45f3-a135-7dd1148ea74e" containerName="limitador"
Sep 17 14:17:39.954666 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:39.954640    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"
Sep 17 14:17:39.959545 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:39.959516    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 14:17:39.960729 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:39.960227    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:17:39.960729 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:39.960304    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 17 14:17:39.962749 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:39.961819    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kc59k\""
Sep 17 14:17:39.967918 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:39.967887    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"]
Sep 17 14:17:40.120893 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:40.120857    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnnj7\" (UniqueName: \"kubernetes.io/projected/ceab7c52-fb1e-499e-bf5f-7081684a0558-kube-api-access-mnnj7\") pod \"postgres-69fb4594cf-9zdbz\" (UID: \"ceab7c52-fb1e-499e-bf5f-7081684a0558\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"
Sep 17 14:17:40.120893 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:40.120894    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ceab7c52-fb1e-499e-bf5f-7081684a0558-data\") pod \"postgres-69fb4594cf-9zdbz\" (UID: \"ceab7c52-fb1e-499e-bf5f-7081684a0558\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"
Sep 17 14:17:40.222165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:40.222085    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnnj7\" (UniqueName: \"kubernetes.io/projected/ceab7c52-fb1e-499e-bf5f-7081684a0558-kube-api-access-mnnj7\") pod \"postgres-69fb4594cf-9zdbz\" (UID: \"ceab7c52-fb1e-499e-bf5f-7081684a0558\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"
Sep 17 14:17:40.222165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:40.222127    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ceab7c52-fb1e-499e-bf5f-7081684a0558-data\") pod \"postgres-69fb4594cf-9zdbz\" (UID: \"ceab7c52-fb1e-499e-bf5f-7081684a0558\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"
Sep 17 14:17:40.222525 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:40.222505    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ceab7c52-fb1e-499e-bf5f-7081684a0558-data\") pod \"postgres-69fb4594cf-9zdbz\" (UID: \"ceab7c52-fb1e-499e-bf5f-7081684a0558\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"
Sep 17 14:17:40.231779 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:40.231754    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnnj7\" (UniqueName: \"kubernetes.io/projected/ceab7c52-fb1e-499e-bf5f-7081684a0558-kube-api-access-mnnj7\") pod \"postgres-69fb4594cf-9zdbz\" (UID: \"ceab7c52-fb1e-499e-bf5f-7081684a0558\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"
Sep 17 14:17:40.244928 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:40.244902    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0ade81b-998b-45f3-a135-7dd1148ea74e" path="/var/lib/kubelet/pods/c0ade81b-998b-45f3-a135-7dd1148ea74e/volumes"
Sep 17 14:17:40.268422 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:40.268400    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"
Sep 17 14:17:40.541477 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:40.541401    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"]
Sep 17 14:17:40.544487 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:17:40.544462    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceab7c52_fb1e_499e_bf5f_7081684a0558.slice/crio-45f51ee557048c46ebd0fec623c76b89809a74616367228f025062cc50ff6693 WatchSource:0}: Error finding container 45f51ee557048c46ebd0fec623c76b89809a74616367228f025062cc50ff6693: Status 404 returned error can't find the container with id 45f51ee557048c46ebd0fec623c76b89809a74616367228f025062cc50ff6693
Sep 17 14:17:40.650796 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:40.650758    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz" event={"ID":"ceab7c52-fb1e-499e-bf5f-7081684a0558","Type":"ContainerStarted","Data":"45f51ee557048c46ebd0fec623c76b89809a74616367228f025062cc50ff6693"}
Sep 17 14:17:45.682349 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:45.682316    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz" event={"ID":"ceab7c52-fb1e-499e-bf5f-7081684a0558","Type":"ContainerStarted","Data":"a366b95c752a5d62ffb79829accd7c93f4c462e10b2cd55190946aee4fb13f98"}
Sep 17 14:17:45.682705 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:45.682411    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"
Sep 17 14:17:51.714911 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:51.714883    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz"
Sep 17 14:17:51.742394 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:51.742343    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zdbz" podStartSLOduration=7.938093482 podStartE2EDuration="12.742326614s" totalImagesPullingTime="4.804233132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:17:45.708834277 +0000 UTC m=+669.999948138" watchObservedRunningTime="2026-09-17 14:17:51.742326614 +0000 UTC m=+676.033440511"
Sep 17 14:17:52.491240 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.491210    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xlskr"]
Sep 17 14:17:52.496118 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.496100    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xlskr"
Sep 17 14:17:52.500210 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.500155    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6svbb\""
Sep 17 14:17:52.501851 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.501832    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 14:17:52.515931 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.515896    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bclqs\" (UniqueName: \"kubernetes.io/projected/a19e4306-9f5e-48bb-9ec8-3d69840a611d-kube-api-access-bclqs\") pod \"authorino-6d4dfc649f-xlskr\" (UID: \"a19e4306-9f5e-48bb-9ec8-3d69840a611d\") " pod="kuadrant-system/authorino-6d4dfc649f-xlskr"
Sep 17 14:17:52.516077 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.515944    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a19e4306-9f5e-48bb-9ec8-3d69840a611d-tls-cert\") pod \"authorino-6d4dfc649f-xlskr\" (UID: \"a19e4306-9f5e-48bb-9ec8-3d69840a611d\") " pod="kuadrant-system/authorino-6d4dfc649f-xlskr"
Sep 17 14:17:52.517636 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.517613    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xlskr"]
Sep 17 14:17:52.616515 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.616473    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bclqs\" (UniqueName: \"kubernetes.io/projected/a19e4306-9f5e-48bb-9ec8-3d69840a611d-kube-api-access-bclqs\") pod \"authorino-6d4dfc649f-xlskr\" (UID: \"a19e4306-9f5e-48bb-9ec8-3d69840a611d\") " pod="kuadrant-system/authorino-6d4dfc649f-xlskr"
Sep 17 14:17:52.616689 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.616527    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a19e4306-9f5e-48bb-9ec8-3d69840a611d-tls-cert\") pod \"authorino-6d4dfc649f-xlskr\" (UID: \"a19e4306-9f5e-48bb-9ec8-3d69840a611d\") " pod="kuadrant-system/authorino-6d4dfc649f-xlskr"
Sep 17 14:17:52.618901 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.618884    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a19e4306-9f5e-48bb-9ec8-3d69840a611d-tls-cert\") pod \"authorino-6d4dfc649f-xlskr\" (UID: \"a19e4306-9f5e-48bb-9ec8-3d69840a611d\") " pod="kuadrant-system/authorino-6d4dfc649f-xlskr"
Sep 17 14:17:52.649857 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.649832    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bclqs\" (UniqueName: \"kubernetes.io/projected/a19e4306-9f5e-48bb-9ec8-3d69840a611d-kube-api-access-bclqs\") pod \"authorino-6d4dfc649f-xlskr\" (UID: \"a19e4306-9f5e-48bb-9ec8-3d69840a611d\") " pod="kuadrant-system/authorino-6d4dfc649f-xlskr"
Sep 17 14:17:52.652831 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.652807    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xlskr"]
Sep 17 14:17:52.653025 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:52.653014    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xlskr"
Sep 17 14:17:53.005699 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:53.005595    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xlskr"]
Sep 17 14:17:53.009416 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:17:53.009379    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda19e4306_9f5e_48bb_9ec8_3d69840a611d.slice/crio-a032584daa8de64dd7b517c8410b01cdba8422b87246364d8216b3657ae4e09e WatchSource:0}: Error finding container a032584daa8de64dd7b517c8410b01cdba8422b87246364d8216b3657ae4e09e: Status 404 returned error can't find the container with id a032584daa8de64dd7b517c8410b01cdba8422b87246364d8216b3657ae4e09e
Sep 17 14:17:53.718432 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:53.718390    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-xlskr" event={"ID":"a19e4306-9f5e-48bb-9ec8-3d69840a611d","Type":"ContainerStarted","Data":"a032584daa8de64dd7b517c8410b01cdba8422b87246364d8216b3657ae4e09e"}
Sep 17 14:17:56.732680 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:56.732637    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-xlskr" event={"ID":"a19e4306-9f5e-48bb-9ec8-3d69840a611d","Type":"ContainerStarted","Data":"0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6"}
Sep 17 14:17:56.733153 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:56.732707    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d4dfc649f-xlskr" podUID="a19e4306-9f5e-48bb-9ec8-3d69840a611d" containerName="authorino" containerID="cri-o://0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6" gracePeriod=30
Sep 17 14:17:56.755587 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:56.755545    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d4dfc649f-xlskr" podStartSLOduration=1.4287198079999999 podStartE2EDuration="4.755530663s" totalImagesPullingTime="3.326810855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:17:56.75520932 +0000 UTC m=+681.046323182" watchObservedRunningTime="2026-09-17 14:17:56.755530663 +0000 UTC m=+681.046644524"
Sep 17 14:17:56.982128 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:56.982104    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xlskr"
Sep 17 14:17:57.046324 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.046246    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bclqs\" (UniqueName: \"kubernetes.io/projected/a19e4306-9f5e-48bb-9ec8-3d69840a611d-kube-api-access-bclqs\") pod \"a19e4306-9f5e-48bb-9ec8-3d69840a611d\" (UID: \"a19e4306-9f5e-48bb-9ec8-3d69840a611d\") "
Sep 17 14:17:57.046324 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.046312    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a19e4306-9f5e-48bb-9ec8-3d69840a611d-tls-cert\") pod \"a19e4306-9f5e-48bb-9ec8-3d69840a611d\" (UID: \"a19e4306-9f5e-48bb-9ec8-3d69840a611d\") "
Sep 17 14:17:57.048230 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.048198    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19e4306-9f5e-48bb-9ec8-3d69840a611d-kube-api-access-bclqs" (OuterVolumeSpecName: "kube-api-access-bclqs") pod "a19e4306-9f5e-48bb-9ec8-3d69840a611d" (UID: "a19e4306-9f5e-48bb-9ec8-3d69840a611d"). InnerVolumeSpecName "kube-api-access-bclqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:17:57.055401 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.055375    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19e4306-9f5e-48bb-9ec8-3d69840a611d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a19e4306-9f5e-48bb-9ec8-3d69840a611d" (UID: "a19e4306-9f5e-48bb-9ec8-3d69840a611d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:17:57.146968 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.146937    2537 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a19e4306-9f5e-48bb-9ec8-3d69840a611d-tls-cert\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:17:57.146968 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.146969    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bclqs\" (UniqueName: \"kubernetes.io/projected/a19e4306-9f5e-48bb-9ec8-3d69840a611d-kube-api-access-bclqs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:17:57.737757 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.737720    2537 generic.go:356] "Generic (PLEG): container finished" podID="a19e4306-9f5e-48bb-9ec8-3d69840a611d" containerID="0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6" exitCode=0
Sep 17 14:17:57.738328 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.737767    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xlskr"
Sep 17 14:17:57.738328 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.737805    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-xlskr" event={"ID":"a19e4306-9f5e-48bb-9ec8-3d69840a611d","Type":"ContainerDied","Data":"0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6"}
Sep 17 14:17:57.738328 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.737848    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-xlskr" event={"ID":"a19e4306-9f5e-48bb-9ec8-3d69840a611d","Type":"ContainerDied","Data":"a032584daa8de64dd7b517c8410b01cdba8422b87246364d8216b3657ae4e09e"}
Sep 17 14:17:57.738328 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.737865    2537 scope.go:117] "RemoveContainer" containerID="0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6"
Sep 17 14:17:57.751660 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.751637    2537 scope.go:117] "RemoveContainer" containerID="0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6"
Sep 17 14:17:57.751940 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:17:57.751919    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6\": container with ID starting with 0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6 not found: ID does not exist" containerID="0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6"
Sep 17 14:17:57.751987 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.751947    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6"} err="failed to get container status \"0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6\": rpc error: code = NotFound desc = could not find container \"0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6\": container with ID starting with 0b08a7d402d8e6bd2cb3cb264a13fa3a1dc92a2def2e4172dd77fd3a79dce6c6 not found: ID does not exist"
Sep 17 14:17:57.764843 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.764814    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xlskr"]
Sep 17 14:17:57.768202 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:57.768163    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xlskr"]
Sep 17 14:17:58.244584 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:58.244542    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a19e4306-9f5e-48bb-9ec8-3d69840a611d" path="/var/lib/kubelet/pods/a19e4306-9f5e-48bb-9ec8-3d69840a611d/volumes"
Sep 17 14:17:59.729064 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.729029    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-647b66497b-zpjhp"]
Sep 17 14:17:59.729469 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.729321    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19e4306-9f5e-48bb-9ec8-3d69840a611d" containerName="authorino"
Sep 17 14:17:59.729469 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.729332    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19e4306-9f5e-48bb-9ec8-3d69840a611d" containerName="authorino"
Sep 17 14:17:59.729469 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.729396    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19e4306-9f5e-48bb-9ec8-3d69840a611d" containerName="authorino"
Sep 17 14:17:59.732082 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.732065    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:17:59.739298 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.739267    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 14:17:59.740095 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.740075    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 14:17:59.740210 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.740123    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 14:17:59.740210 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.740150    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hnczm\""
Sep 17 14:17:59.754110 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.754086    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-647b66497b-zpjhp"]
Sep 17 14:17:59.767482 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.767458    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-webhook-cert\") pod \"maas-controller-647b66497b-zpjhp\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") " pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:17:59.767603 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.767491    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8k6\" (UniqueName: \"kubernetes.io/projected/74d8e18c-ecff-4273-8d39-b156ee18ec07-kube-api-access-7c8k6\") pod \"maas-controller-647b66497b-zpjhp\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") " pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:17:59.767603 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.767557    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-metrics-certs\") pod \"maas-controller-647b66497b-zpjhp\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") " pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:17:59.868650 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.868615    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-metrics-certs\") pod \"maas-controller-647b66497b-zpjhp\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") " pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:17:59.868834 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.868668    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-webhook-cert\") pod \"maas-controller-647b66497b-zpjhp\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") " pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:17:59.868834 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:17:59.868764    2537 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 17 14:17:59.868834 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.868788    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8k6\" (UniqueName: \"kubernetes.io/projected/74d8e18c-ecff-4273-8d39-b156ee18ec07-kube-api-access-7c8k6\") pod \"maas-controller-647b66497b-zpjhp\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") " pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:17:59.868834 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:17:59.868833    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-metrics-certs podName:74d8e18c-ecff-4273-8d39-b156ee18ec07 nodeName:}" failed. No retries permitted until 2026-09-17 14:18:00.368814593 +0000 UTC m=+684.659928445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-metrics-certs") pod "maas-controller-647b66497b-zpjhp" (UID: "74d8e18c-ecff-4273-8d39-b156ee18ec07") : secret "maas-controller-metrics-tls" not found
Sep 17 14:17:59.871135 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.871110    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-webhook-cert\") pod \"maas-controller-647b66497b-zpjhp\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") " pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:17:59.882367 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.882347    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8k6\" (UniqueName: \"kubernetes.io/projected/74d8e18c-ecff-4273-8d39-b156ee18ec07-kube-api-access-7c8k6\") pod \"maas-controller-647b66497b-zpjhp\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") " pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:17:59.949966 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.949938    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-4b4sj"]
Sep 17 14:17:59.952280 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.952263    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:17:59.969665 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.969640    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-metrics-certs\") pod \"maas-controller-6bf6d9dbbc-4b4sj\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") " pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:17:59.969780 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.969669    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p28pg\" (UniqueName: \"kubernetes.io/projected/7b259c42-97a1-49f2-a4b7-48631ebc0a08-kube-api-access-p28pg\") pod \"maas-controller-6bf6d9dbbc-4b4sj\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") " pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:17:59.969780 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.969716    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-webhook-cert\") pod \"maas-controller-6bf6d9dbbc-4b4sj\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") " pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:17:59.981524 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:17:59.981463    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-4b4sj"]
Sep 17 14:18:00.070974 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.070935    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-webhook-cert\") pod \"maas-controller-6bf6d9dbbc-4b4sj\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") " pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:18:00.071140 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.070995    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-metrics-certs\") pod \"maas-controller-6bf6d9dbbc-4b4sj\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") " pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:18:00.071140 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.071015    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p28pg\" (UniqueName: \"kubernetes.io/projected/7b259c42-97a1-49f2-a4b7-48631ebc0a08-kube-api-access-p28pg\") pod \"maas-controller-6bf6d9dbbc-4b4sj\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") " pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:18:00.071140 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:18:00.071094    2537 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 17 14:18:00.071303 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:18:00.071156    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-metrics-certs podName:7b259c42-97a1-49f2-a4b7-48631ebc0a08 nodeName:}" failed. No retries permitted until 2026-09-17 14:18:00.571141488 +0000 UTC m=+684.862255327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-metrics-certs") pod "maas-controller-6bf6d9dbbc-4b4sj" (UID: "7b259c42-97a1-49f2-a4b7-48631ebc0a08") : secret "maas-controller-metrics-tls" not found
Sep 17 14:18:00.073414 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.073396    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-webhook-cert\") pod \"maas-controller-6bf6d9dbbc-4b4sj\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") " pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:18:00.085591 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.085567    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p28pg\" (UniqueName: \"kubernetes.io/projected/7b259c42-97a1-49f2-a4b7-48631ebc0a08-kube-api-access-p28pg\") pod \"maas-controller-6bf6d9dbbc-4b4sj\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") " pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:18:00.373453 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.373412    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-metrics-certs\") pod \"maas-controller-647b66497b-zpjhp\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") " pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:18:00.375822 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.375795    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-metrics-certs\") pod \"maas-controller-647b66497b-zpjhp\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") " pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:18:00.574864 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.574824    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-metrics-certs\") pod \"maas-controller-6bf6d9dbbc-4b4sj\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") " pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:18:00.577154 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.577129    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-metrics-certs\") pod \"maas-controller-6bf6d9dbbc-4b4sj\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") " pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:18:00.643072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.642997    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:18:00.862318 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.862284    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:18:00.929410 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:00.929362    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-647b66497b-zpjhp"]
Sep 17 14:18:00.934118 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:18:00.934084    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d8e18c_ecff_4273_8d39_b156ee18ec07.slice/crio-ea3d7144a0942667dd7037b38d01140dc3d2d1354039b2a13c29c80f5e606c91 WatchSource:0}: Error finding container ea3d7144a0942667dd7037b38d01140dc3d2d1354039b2a13c29c80f5e606c91: Status 404 returned error can't find the container with id ea3d7144a0942667dd7037b38d01140dc3d2d1354039b2a13c29c80f5e606c91
Sep 17 14:18:01.141297 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:01.141274    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-4b4sj"]
Sep 17 14:18:01.143597 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:18:01.143564    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b259c42_97a1_49f2_a4b7_48631ebc0a08.slice/crio-3852820e51d97efd6932aa590033db1646dfaa3d61ca7eb60ba540c79e0fa13c WatchSource:0}: Error finding container 3852820e51d97efd6932aa590033db1646dfaa3d61ca7eb60ba540c79e0fa13c: Status 404 returned error can't find the container with id 3852820e51d97efd6932aa590033db1646dfaa3d61ca7eb60ba540c79e0fa13c
Sep 17 14:18:01.759705 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:01.759654    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-647b66497b-zpjhp" event={"ID":"74d8e18c-ecff-4273-8d39-b156ee18ec07","Type":"ContainerStarted","Data":"ea3d7144a0942667dd7037b38d01140dc3d2d1354039b2a13c29c80f5e606c91"}
Sep 17 14:18:01.761496 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:01.761461    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj" event={"ID":"7b259c42-97a1-49f2-a4b7-48631ebc0a08","Type":"ContainerStarted","Data":"3852820e51d97efd6932aa590033db1646dfaa3d61ca7eb60ba540c79e0fa13c"}
Sep 17 14:18:03.772380 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:03.772339    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj" event={"ID":"7b259c42-97a1-49f2-a4b7-48631ebc0a08","Type":"ContainerStarted","Data":"e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94"}
Sep 17 14:18:03.772816 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:03.772458    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:18:03.774435 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:03.774407    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-647b66497b-zpjhp" event={"ID":"74d8e18c-ecff-4273-8d39-b156ee18ec07","Type":"ContainerStarted","Data":"4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245"}
Sep 17 14:18:03.774555 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:03.774540    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:18:03.799480 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:03.799429    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj" podStartSLOduration=2.340767329 podStartE2EDuration="4.79941185s" totalImagesPullingTime="2.458644521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:03.798334728 +0000 UTC m=+688.089448590" watchObservedRunningTime="2026-09-17 14:18:03.79941185 +0000 UTC m=+688.090525709"
Sep 17 14:18:03.825798 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:03.825748    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-647b66497b-zpjhp" podStartSLOduration=2.165043627 podStartE2EDuration="4.825731093s" totalImagesPullingTime="2.660687466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:03.824532037 +0000 UTC m=+688.115645900" watchObservedRunningTime="2026-09-17 14:18:03.825731093 +0000 UTC m=+688.116844953"
Sep 17 14:18:14.782129 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:14.782095    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:18:14.782533 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:14.782169    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:18:14.879129 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:14.879098    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-647b66497b-zpjhp"]
Sep 17 14:18:14.879350 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:14.879304    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-647b66497b-zpjhp" podUID="74d8e18c-ecff-4273-8d39-b156ee18ec07" containerName="manager" containerID="cri-o://4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245" gracePeriod=30
Sep 17 14:18:15.129777 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.129751    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:18:15.178735 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.178699    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c8k6\" (UniqueName: \"kubernetes.io/projected/74d8e18c-ecff-4273-8d39-b156ee18ec07-kube-api-access-7c8k6\") pod \"74d8e18c-ecff-4273-8d39-b156ee18ec07\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") "
Sep 17 14:18:15.178910 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.178744    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-webhook-cert\") pod \"74d8e18c-ecff-4273-8d39-b156ee18ec07\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") "
Sep 17 14:18:15.178910 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.178778    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-metrics-certs\") pod \"74d8e18c-ecff-4273-8d39-b156ee18ec07\" (UID: \"74d8e18c-ecff-4273-8d39-b156ee18ec07\") "
Sep 17 14:18:15.180895 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.180867    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "74d8e18c-ecff-4273-8d39-b156ee18ec07" (UID: "74d8e18c-ecff-4273-8d39-b156ee18ec07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:18:15.180895 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.180885    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "74d8e18c-ecff-4273-8d39-b156ee18ec07" (UID: "74d8e18c-ecff-4273-8d39-b156ee18ec07"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:18:15.181040 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.180922    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d8e18c-ecff-4273-8d39-b156ee18ec07-kube-api-access-7c8k6" (OuterVolumeSpecName: "kube-api-access-7c8k6") pod "74d8e18c-ecff-4273-8d39-b156ee18ec07" (UID: "74d8e18c-ecff-4273-8d39-b156ee18ec07"). InnerVolumeSpecName "kube-api-access-7c8k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:18:15.279318 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.279285    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c8k6\" (UniqueName: \"kubernetes.io/projected/74d8e18c-ecff-4273-8d39-b156ee18ec07-kube-api-access-7c8k6\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:18:15.279318 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.279315    2537 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-webhook-cert\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:18:15.279318 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.279324    2537 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d8e18c-ecff-4273-8d39-b156ee18ec07-metrics-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:18:15.827894 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.827852    2537 generic.go:356] "Generic (PLEG): container finished" podID="74d8e18c-ecff-4273-8d39-b156ee18ec07" containerID="4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245" exitCode=0
Sep 17 14:18:15.828410 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.827918    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-647b66497b-zpjhp"
Sep 17 14:18:15.828410 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.827933    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-647b66497b-zpjhp" event={"ID":"74d8e18c-ecff-4273-8d39-b156ee18ec07","Type":"ContainerDied","Data":"4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245"}
Sep 17 14:18:15.828410 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.828008    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-647b66497b-zpjhp" event={"ID":"74d8e18c-ecff-4273-8d39-b156ee18ec07","Type":"ContainerDied","Data":"ea3d7144a0942667dd7037b38d01140dc3d2d1354039b2a13c29c80f5e606c91"}
Sep 17 14:18:15.828410 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.828029    2537 scope.go:117] "RemoveContainer" containerID="4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245"
Sep 17 14:18:15.836430 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.836412    2537 scope.go:117] "RemoveContainer" containerID="4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245"
Sep 17 14:18:15.836702 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:18:15.836683    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245\": container with ID starting with 4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245 not found: ID does not exist" containerID="4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245"
Sep 17 14:18:15.836756 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.836710    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245"} err="failed to get container status \"4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245\": rpc error: code = NotFound desc = could not find container \"4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245\": container with ID starting with 4d85853de417d3b0c8e162783d6c7049d6e4620b44339544ebea2250b6cdf245 not found: ID does not exist"
Sep 17 14:18:15.851009 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.850980    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-647b66497b-zpjhp"]
Sep 17 14:18:15.857707 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:15.857684    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-647b66497b-zpjhp"]
Sep 17 14:18:16.244418 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:16.244384    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74d8e18c-ecff-4273-8d39-b156ee18ec07" path="/var/lib/kubelet/pods/74d8e18c-ecff-4273-8d39-b156ee18ec07/volumes"
Sep 17 14:18:37.209881 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.209843    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"]
Sep 17 14:18:37.210326 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.210172    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74d8e18c-ecff-4273-8d39-b156ee18ec07" containerName="manager"
Sep 17 14:18:37.210326 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.210214    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="74d8e18c-ecff-4273-8d39-b156ee18ec07" containerName="manager"
Sep 17 14:18:37.210326 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.210262    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="74d8e18c-ecff-4273-8d39-b156ee18ec07" containerName="manager"
Sep 17 14:18:37.216759 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.216731    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.221299 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.221272    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 17 14:18:37.222569 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.222549    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2n86r\""
Sep 17 14:18:37.222660 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.222548    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 14:18:37.229877 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.229844    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"]
Sep 17 14:18:37.240499 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.240468    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-maas-api-tls\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.240676 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.240509    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fd9w\" (UniqueName: \"kubernetes.io/projected/66670616-aaa0-4758-a736-446f4e654409-kube-api-access-6fd9w\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.240676 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.240587    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66670616-aaa0-4758-a736-446f4e654409-openshift-service-ca\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.240676 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.240652    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-metrics-certs\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.341459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.341412    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-maas-api-tls\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.341459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.341465    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fd9w\" (UniqueName: \"kubernetes.io/projected/66670616-aaa0-4758-a736-446f4e654409-kube-api-access-6fd9w\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.341680 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.341509    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66670616-aaa0-4758-a736-446f4e654409-openshift-service-ca\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.341680 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.341554    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-metrics-certs\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.341680 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:18:37.341560    2537 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 17 14:18:37.341680 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:18:37.341621    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-maas-api-tls podName:66670616-aaa0-4758-a736-446f4e654409 nodeName:}" failed. No retries permitted until 2026-09-17 14:18:37.841604963 +0000 UTC m=+722.132718802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-maas-api-tls") pod "maas-api-6c9b6644d-r8s7s" (UID: "66670616-aaa0-4758-a736-446f4e654409") : secret "maas-api-serving-cert" not found
Sep 17 14:18:37.342274 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.342255    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66670616-aaa0-4758-a736-446f4e654409-openshift-service-ca\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.344081 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.344058    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-metrics-certs\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.352597 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.352570    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fd9w\" (UniqueName: \"kubernetes.io/projected/66670616-aaa0-4758-a736-446f4e654409-kube-api-access-6fd9w\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.806118 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.806083    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"]
Sep 17 14:18:37.810389 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.810372    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"
Sep 17 14:18:37.813095 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.813074    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 17 14:18:37.818111 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.818087    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"]
Sep 17 14:18:37.821096 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.821080    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"
Sep 17 14:18:37.828641 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.828617    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"]
Sep 17 14:18:37.837404 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.837375    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"]
Sep 17 14:18:37.846316 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.846283    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmtg\" (UniqueName: \"kubernetes.io/projected/f5f4a040-2239-41eb-b7c7-c941c7a3e30c-kube-api-access-vcmtg\") pod \"payload-processing-5b54b7fbb7-fkwcw\" (UID: \"f5f4a040-2239-41eb-b7c7-c941c7a3e30c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"
Sep 17 14:18:37.846500 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.846359    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-maas-api-tls\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.846500 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.846397    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9df4506-8383-4582-92b0-b8c06c843697-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-q95h6\" (UID: \"c9df4506-8383-4582-92b0-b8c06c843697\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"
Sep 17 14:18:37.846500 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.846436    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f4a040-2239-41eb-b7c7-c941c7a3e30c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fkwcw\" (UID: \"f5f4a040-2239-41eb-b7c7-c941c7a3e30c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"
Sep 17 14:18:37.846500 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.846464    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jhm2\" (UniqueName: \"kubernetes.io/projected/c9df4506-8383-4582-92b0-b8c06c843697-kube-api-access-5jhm2\") pod \"payload-pre-processing-59fbc47bf8-q95h6\" (UID: \"c9df4506-8383-4582-92b0-b8c06c843697\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"
Sep 17 14:18:37.849696 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.849656    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-maas-api-tls\") pod \"maas-api-6c9b6644d-r8s7s\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") " pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:37.947456 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.947418    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9df4506-8383-4582-92b0-b8c06c843697-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-q95h6\" (UID: \"c9df4506-8383-4582-92b0-b8c06c843697\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"
Sep 17 14:18:37.947675 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.947481    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f4a040-2239-41eb-b7c7-c941c7a3e30c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fkwcw\" (UID: \"f5f4a040-2239-41eb-b7c7-c941c7a3e30c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"
Sep 17 14:18:37.947675 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.947513    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhm2\" (UniqueName: \"kubernetes.io/projected/c9df4506-8383-4582-92b0-b8c06c843697-kube-api-access-5jhm2\") pod \"payload-pre-processing-59fbc47bf8-q95h6\" (UID: \"c9df4506-8383-4582-92b0-b8c06c843697\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"
Sep 17 14:18:37.947675 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.947567    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmtg\" (UniqueName: \"kubernetes.io/projected/f5f4a040-2239-41eb-b7c7-c941c7a3e30c-kube-api-access-vcmtg\") pod \"payload-processing-5b54b7fbb7-fkwcw\" (UID: \"f5f4a040-2239-41eb-b7c7-c941c7a3e30c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"
Sep 17 14:18:37.948303 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.948281    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9df4506-8383-4582-92b0-b8c06c843697-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-q95h6\" (UID: \"c9df4506-8383-4582-92b0-b8c06c843697\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"
Sep 17 14:18:37.948358 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.948289    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f4a040-2239-41eb-b7c7-c941c7a3e30c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fkwcw\" (UID: \"f5f4a040-2239-41eb-b7c7-c941c7a3e30c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"
Sep 17 14:18:37.965778 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.965741    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmtg\" (UniqueName: \"kubernetes.io/projected/f5f4a040-2239-41eb-b7c7-c941c7a3e30c-kube-api-access-vcmtg\") pod \"payload-processing-5b54b7fbb7-fkwcw\" (UID: \"f5f4a040-2239-41eb-b7c7-c941c7a3e30c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"
Sep 17 14:18:37.966604 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:37.966582    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhm2\" (UniqueName: \"kubernetes.io/projected/c9df4506-8383-4582-92b0-b8c06c843697-kube-api-access-5jhm2\") pod \"payload-pre-processing-59fbc47bf8-q95h6\" (UID: \"c9df4506-8383-4582-92b0-b8c06c843697\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"
Sep 17 14:18:38.120859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:38.120818    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"
Sep 17 14:18:38.126920 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:38.126901    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:38.131923 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:38.131896    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"
Sep 17 14:18:38.427968 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:38.427694    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"]
Sep 17 14:18:38.429825 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:18:38.429793    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66670616_aaa0_4758_a736_446f4e654409.slice/crio-79457e04a92cd4b0afb83e7e63bfc80b9460649ea7055c59fff90d0f5d15a2e1 WatchSource:0}: Error finding container 79457e04a92cd4b0afb83e7e63bfc80b9460649ea7055c59fff90d0f5d15a2e1: Status 404 returned error can't find the container with id 79457e04a92cd4b0afb83e7e63bfc80b9460649ea7055c59fff90d0f5d15a2e1
Sep 17 14:18:38.429947 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:38.429931    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"]
Sep 17 14:18:38.430846 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:18:38.430820    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9df4506_8383_4582_92b0_b8c06c843697.slice/crio-20312f0b766a07d2e86fb9d742ab11e1fa18888e66ae7a2cd267dc4b94e51035 WatchSource:0}: Error finding container 20312f0b766a07d2e86fb9d742ab11e1fa18888e66ae7a2cd267dc4b94e51035: Status 404 returned error can't find the container with id 20312f0b766a07d2e86fb9d742ab11e1fa18888e66ae7a2cd267dc4b94e51035
Sep 17 14:18:38.431964 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:38.431943    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"]
Sep 17 14:18:38.432708 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:18:38.432685    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f4a040_2239_41eb_b7c7_c941c7a3e30c.slice/crio-c5d170c907d774a7f2be78f25c234713827c5b4f08193fd07a2697bd0907be24 WatchSource:0}: Error finding container c5d170c907d774a7f2be78f25c234713827c5b4f08193fd07a2697bd0907be24: Status 404 returned error can't find the container with id c5d170c907d774a7f2be78f25c234713827c5b4f08193fd07a2697bd0907be24
Sep 17 14:18:38.926348 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:38.926311    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw" event={"ID":"f5f4a040-2239-41eb-b7c7-c941c7a3e30c","Type":"ContainerStarted","Data":"c5d170c907d774a7f2be78f25c234713827c5b4f08193fd07a2697bd0907be24"}
Sep 17 14:18:38.927704 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:38.927675    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s" event={"ID":"66670616-aaa0-4758-a736-446f4e654409","Type":"ContainerStarted","Data":"79457e04a92cd4b0afb83e7e63bfc80b9460649ea7055c59fff90d0f5d15a2e1"}
Sep 17 14:18:38.928988 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:38.928968    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6" event={"ID":"c9df4506-8383-4582-92b0-b8c06c843697","Type":"ContainerStarted","Data":"20312f0b766a07d2e86fb9d742ab11e1fa18888e66ae7a2cd267dc4b94e51035"}
Sep 17 14:18:40.941167 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:40.941117    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw" event={"ID":"f5f4a040-2239-41eb-b7c7-c941c7a3e30c","Type":"ContainerStarted","Data":"8728460cdedbfd38eb364c2a031a22d42298a22bdd97b3af295c97a9ab7f1bdb"}
Sep 17 14:18:40.941646 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:40.941324    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"
Sep 17 14:18:40.942632 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:40.942602    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s" event={"ID":"66670616-aaa0-4758-a736-446f4e654409","Type":"ContainerStarted","Data":"9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f"}
Sep 17 14:18:40.942752 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:40.942649    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:40.944065 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:40.944043    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6" event={"ID":"c9df4506-8383-4582-92b0-b8c06c843697","Type":"ContainerStarted","Data":"7bd0d2536fdb8a101a90a679971811cb60a1213631a4af36f32ac85ec4362b10"}
Sep 17 14:18:40.944168 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:40.944093    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"
Sep 17 14:18:40.962058 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:40.962015    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw" podStartSLOduration=2.479452386 podStartE2EDuration="3.962001866s" totalImagesPullingTime="1.48254948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:40.960682 +0000 UTC m=+725.251795862" watchObservedRunningTime="2026-09-17 14:18:40.962001866 +0000 UTC m=+725.253115756"
Sep 17 14:18:41.008452 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:41.008390    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s" podStartSLOduration=2.033792923 podStartE2EDuration="4.008376438s" totalImagesPullingTime="1.974583515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:41.006973567 +0000 UTC m=+725.298087431" watchObservedRunningTime="2026-09-17 14:18:41.008376438 +0000 UTC m=+725.299490300"
Sep 17 14:18:41.008634 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:41.008505    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6" podStartSLOduration=2.527107187 podStartE2EDuration="4.008496022s" totalImagesPullingTime="1.481388835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:40.982743122 +0000 UTC m=+725.273856982" watchObservedRunningTime="2026-09-17 14:18:41.008496022 +0000 UTC m=+725.299609884"
Sep 17 14:18:46.955638 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:46.955603    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:18:51.952011 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:51.951978    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-q95h6"
Sep 17 14:18:51.952473 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:18:51.952062    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-fkwcw"
Sep 17 14:19:44.714574 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.714536    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"]
Sep 17 14:19:44.718273 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.718249    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.722831 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.722801    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 17 14:19:44.722831 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.722818    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 14:19:44.723034 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.722821    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 14:19:44.723034 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.722871    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pqctc\""
Sep 17 14:19:44.730941 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.730915    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"]
Sep 17 14:19:44.783951 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.783916    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8tst\" (UniqueName: \"kubernetes.io/projected/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-kube-api-access-p8tst\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.784163 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.784023    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.784163 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.784058    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.784338 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.784155    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.784338 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.784217    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.784338 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.784241    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.885338 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.885286    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.885338 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.885339    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.885707 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.885356    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.885707 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.885383    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8tst\" (UniqueName: \"kubernetes.io/projected/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-kube-api-access-p8tst\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.885707 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.885422    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.885707 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.885440    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.885890 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.885828    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.885890 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.885867    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.885967 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.885899    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.887765 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.887743    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.887967 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.887951    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:44.896604 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:44.896572    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8tst\" (UniqueName: \"kubernetes.io/projected/d8328e56-c364-4b6a-b9f7-2b70765bd1a4-kube-api-access-p8tst\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf\" (UID: \"d8328e56-c364-4b6a-b9f7-2b70765bd1a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:45.028509 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:45.028419    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:45.306447 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:45.306384    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"]
Sep 17 14:19:45.309198 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:19:45.309149    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8328e56_c364_4b6a_b9f7_2b70765bd1a4.slice/crio-367faaec7023d98ea30f1fd358a04f9b1857484e387ee309eb9b80c342c807cc WatchSource:0}: Error finding container 367faaec7023d98ea30f1fd358a04f9b1857484e387ee309eb9b80c342c807cc: Status 404 returned error can't find the container with id 367faaec7023d98ea30f1fd358a04f9b1857484e387ee309eb9b80c342c807cc
Sep 17 14:19:45.310632 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:45.310610    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:19:46.232487 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:46.232438    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf" event={"ID":"d8328e56-c364-4b6a-b9f7-2b70765bd1a4","Type":"ContainerStarted","Data":"367faaec7023d98ea30f1fd358a04f9b1857484e387ee309eb9b80c342c807cc"}
Sep 17 14:19:47.238153 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:47.238111    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf" event={"ID":"d8328e56-c364-4b6a-b9f7-2b70765bd1a4","Type":"ContainerStarted","Data":"c4d4e3607e12ea45ac12624ab8214d95cec3f9d571349dbabdfb0c2a35e33e06"}
Sep 17 14:19:47.263552 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:47.263502    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf" podStartSLOduration=2.247592294 podStartE2EDuration="3.263487853s" totalImagesPullingTime="1.015895559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:47.261875074 +0000 UTC m=+791.552988933" watchObservedRunningTime="2026-09-17 14:19:47.263487853 +0000 UTC m=+791.554601713"
Sep 17 14:19:55.029246 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:55.029205    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:55.029710 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:55.029273    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:55.042037 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:55.042011    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:19:55.285650 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:19:55.285570    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf"
Sep 17 14:20:23.023793 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.023755    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"]
Sep 17 14:20:23.028408 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.028384    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.034364 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.034338    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 17 14:20:23.046945 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.046912    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"]
Sep 17 14:20:23.210085 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.210042    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hhgm\" (UniqueName: \"kubernetes.io/projected/607044ce-9261-46f4-a8ae-d8749056957f-kube-api-access-7hhgm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.210321 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.210091    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.210321 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.210118    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.210321 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.210277    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.210482 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.210333    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.210482 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.210371    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607044ce-9261-46f4-a8ae-d8749056957f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.311735 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.311636    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hhgm\" (UniqueName: \"kubernetes.io/projected/607044ce-9261-46f4-a8ae-d8749056957f-kube-api-access-7hhgm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.311735 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.311682    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.311735 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.311700    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.311996 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.311773    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.311996 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.311842    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.311996 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.311879    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607044ce-9261-46f4-a8ae-d8749056957f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.312244 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.312224    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.312333 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.312306    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.312333 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.312319    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.314012 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.313985    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607044ce-9261-46f4-a8ae-d8749056957f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.314330 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.314313    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607044ce-9261-46f4-a8ae-d8749056957f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.323529 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.323506    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hhgm\" (UniqueName: \"kubernetes.io/projected/607044ce-9261-46f4-a8ae-d8749056957f-kube-api-access-7hhgm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5\" (UID: \"607044ce-9261-46f4-a8ae-d8749056957f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.339392 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.339355    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:23.615697 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:23.615670    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"]
Sep 17 14:20:23.617375 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:20:23.617348    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607044ce_9261_46f4_a8ae_d8749056957f.slice/crio-dba25c1008764694f7ce70bcd11091bb45c160102666d3ec65c1b3207ffa52a5 WatchSource:0}: Error finding container dba25c1008764694f7ce70bcd11091bb45c160102666d3ec65c1b3207ffa52a5: Status 404 returned error can't find the container with id dba25c1008764694f7ce70bcd11091bb45c160102666d3ec65c1b3207ffa52a5
Sep 17 14:20:24.403333 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:24.403298    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5" event={"ID":"607044ce-9261-46f4-a8ae-d8749056957f","Type":"ContainerStarted","Data":"ea6e95dc59a27b4f3e3490095c0159b2493574b5fae8f6aa8aa450b06b4180bb"}
Sep 17 14:20:24.403719 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:24.403355    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5" event={"ID":"607044ce-9261-46f4-a8ae-d8749056957f","Type":"ContainerStarted","Data":"dba25c1008764694f7ce70bcd11091bb45c160102666d3ec65c1b3207ffa52a5"}
Sep 17 14:20:24.433847 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:24.433791    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5" podStartSLOduration=2.222246976 podStartE2EDuration="2.433774149s" totalImagesPullingTime="211.527173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:20:24.432556897 +0000 UTC m=+828.723670757" watchObservedRunningTime="2026-09-17 14:20:24.433774149 +0000 UTC m=+828.724888013"
Sep 17 14:20:33.340206 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:33.340128    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:33.340633 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:33.340232    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:33.352732 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:33.352709    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:33.452287 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:33.452260    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5"
Sep 17 14:20:34.408085 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.408049    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"]
Sep 17 14:20:34.411380 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.411363    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.414339 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.414323    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 17 14:20:34.426032 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.426006    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"]
Sep 17 14:20:34.506352 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.506320    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.506352 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.506355    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.506531 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.506379    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mpzr\" (UniqueName: \"kubernetes.io/projected/ff1c19f2-a335-4432-a231-f61094c134bb-kube-api-access-2mpzr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.506531 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.506407    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.506531 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.506482    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1c19f2-a335-4432-a231-f61094c134bb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.506623 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.506551    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.607354 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.607319    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.607354 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.607355    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.607560 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.607378    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mpzr\" (UniqueName: \"kubernetes.io/projected/ff1c19f2-a335-4432-a231-f61094c134bb-kube-api-access-2mpzr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.607560 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.607411    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.607560 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.607453    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1c19f2-a335-4432-a231-f61094c134bb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.607713 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.607623    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.607862 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.607837    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.608005 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.607881    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.608068 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.608037    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.609783 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.609763    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1c19f2-a335-4432-a231-f61094c134bb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.610047 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.610028    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1c19f2-a335-4432-a231-f61094c134bb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.616571 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.616545    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mpzr\" (UniqueName: \"kubernetes.io/projected/ff1c19f2-a335-4432-a231-f61094c134bb-kube-api-access-2mpzr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k\" (UID: \"ff1c19f2-a335-4432-a231-f61094c134bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:34.720482 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:34.720399    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:35.007504 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:35.007479    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"]
Sep 17 14:20:35.010077 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:20:35.010050    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff1c19f2_a335_4432_a231_f61094c134bb.slice/crio-a870ec4863ed976ca08c0a8b75a186c80360355b4fdeaebfcc08c13a1329ec62 WatchSource:0}: Error finding container a870ec4863ed976ca08c0a8b75a186c80360355b4fdeaebfcc08c13a1329ec62: Status 404 returned error can't find the container with id a870ec4863ed976ca08c0a8b75a186c80360355b4fdeaebfcc08c13a1329ec62
Sep 17 14:20:35.451085 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:35.451042    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k" event={"ID":"ff1c19f2-a335-4432-a231-f61094c134bb","Type":"ContainerStarted","Data":"13a1a14c116d70f2d38454b8283a132da925e53a4e05f196ff2f856cb6da0657"}
Sep 17 14:20:35.451461 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:35.451098    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k" event={"ID":"ff1c19f2-a335-4432-a231-f61094c134bb","Type":"ContainerStarted","Data":"a870ec4863ed976ca08c0a8b75a186c80360355b4fdeaebfcc08c13a1329ec62"}
Sep 17 14:20:35.481570 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:35.481522    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k" podStartSLOduration=1.280931694 podStartE2EDuration="1.481505904s" totalImagesPullingTime="200.57421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:20:35.478769048 +0000 UTC m=+839.769882909" watchObservedRunningTime="2026-09-17 14:20:35.481505904 +0000 UTC m=+839.772619785"
Sep 17 14:20:44.720909 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:44.720866    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:44.721333 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:44.720949    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:44.733440 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:44.733418    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:20:45.507493 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:20:45.507464    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k"
Sep 17 14:21:36.139286 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:36.139218    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:21:36.139788 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:36.139388    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:21:54.819418 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.819388    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"]
Sep 17 14:21:54.823452 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.823397    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:54.827001 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.826978    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-080801-serving-cert\""
Sep 17 14:21:54.837225 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.837155    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"]
Sep 17 14:21:54.887995 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.887956    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:54.888172 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.888002    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-metrics-certs\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:54.888172 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.888123    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgs2p\" (UniqueName: \"kubernetes.io/projected/567841f6-61d0-495d-9d76-efa5f9f00db7-kube-api-access-pgs2p\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:54.888305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.888229    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567841f6-61d0-495d-9d76-efa5f9f00db7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:54.988693 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.988657    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:54.988693 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.988692    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-metrics-certs\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:54.988950 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.988722    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgs2p\" (UniqueName: \"kubernetes.io/projected/567841f6-61d0-495d-9d76-efa5f9f00db7-kube-api-access-pgs2p\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:54.988950 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.988759    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567841f6-61d0-495d-9d76-efa5f9f00db7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:54.988950 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:21:54.988851    2537 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-serving-cert: secret "maas-api-e2e-worker-w0-080801-serving-cert" not found
Sep 17 14:21:54.989107 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:21:54.988964    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-maas-api-tls podName:567841f6-61d0-495d-9d76-efa5f9f00db7 nodeName:}" failed. No retries permitted until 2026-09-17 14:21:55.488941897 +0000 UTC m=+919.780055753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-maas-api-tls") pod "maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59" (UID: "567841f6-61d0-495d-9d76-efa5f9f00db7") : secret "maas-api-e2e-worker-w0-080801-serving-cert" not found
Sep 17 14:21:54.989549 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.989529    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567841f6-61d0-495d-9d76-efa5f9f00db7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:54.991184 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:54.991155    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-metrics-certs\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:55.006553 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.006523    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgs2p\" (UniqueName: \"kubernetes.io/projected/567841f6-61d0-495d-9d76-efa5f9f00db7-kube-api-access-pgs2p\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:55.494223 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.494190    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:55.496861 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.496825    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:55.584722 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.584688    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"]
Sep 17 14:21:55.588339 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.588321    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:21:55.595750 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.595730    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-080801\""
Sep 17 14:21:55.606828 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.606800    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"]
Sep 17 14:21:55.612012 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.611984    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"]
Sep 17 14:21:55.615818 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.615799    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:21:55.618740 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.618581    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-080801-dockercfg-pb6rk\""
Sep 17 14:21:55.638493 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.638467    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"]
Sep 17 14:21:55.695701 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.695660    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca97e45d-8be0-417d-a657-436387f6b59b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr\" (UID: \"ca97e45d-8be0-417d-a657-436387f6b59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:21:55.695701 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.695718    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8qzm\" (UniqueName: \"kubernetes.io/projected/26d972e9-2881-4707-bffd-98bab02b8f41-kube-api-access-w8qzm\") pod \"payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77\" (UID: \"26d972e9-2881-4707-bffd-98bab02b8f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:21:55.695960 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.695839    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6pk5\" (UniqueName: \"kubernetes.io/projected/ca97e45d-8be0-417d-a657-436387f6b59b-kube-api-access-m6pk5\") pod \"payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr\" (UID: \"ca97e45d-8be0-417d-a657-436387f6b59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:21:55.695960 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.695878    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d972e9-2881-4707-bffd-98bab02b8f41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77\" (UID: \"26d972e9-2881-4707-bffd-98bab02b8f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:21:55.734667 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.734637    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:55.797105 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.797058    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qzm\" (UniqueName: \"kubernetes.io/projected/26d972e9-2881-4707-bffd-98bab02b8f41-kube-api-access-w8qzm\") pod \"payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77\" (UID: \"26d972e9-2881-4707-bffd-98bab02b8f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:21:55.797241 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.797205    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6pk5\" (UniqueName: \"kubernetes.io/projected/ca97e45d-8be0-417d-a657-436387f6b59b-kube-api-access-m6pk5\") pod \"payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr\" (UID: \"ca97e45d-8be0-417d-a657-436387f6b59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:21:55.797329 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.797239    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d972e9-2881-4707-bffd-98bab02b8f41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77\" (UID: \"26d972e9-2881-4707-bffd-98bab02b8f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:21:55.797329 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.797297    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca97e45d-8be0-417d-a657-436387f6b59b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr\" (UID: \"ca97e45d-8be0-417d-a657-436387f6b59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:21:55.798668 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.798605    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca97e45d-8be0-417d-a657-436387f6b59b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr\" (UID: \"ca97e45d-8be0-417d-a657-436387f6b59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:21:55.798969 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.798944    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d972e9-2881-4707-bffd-98bab02b8f41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77\" (UID: \"26d972e9-2881-4707-bffd-98bab02b8f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:21:55.809071 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.809036    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6pk5\" (UniqueName: \"kubernetes.io/projected/ca97e45d-8be0-417d-a657-436387f6b59b-kube-api-access-m6pk5\") pod \"payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr\" (UID: \"ca97e45d-8be0-417d-a657-436387f6b59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:21:55.811542 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.811510    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8qzm\" (UniqueName: \"kubernetes.io/projected/26d972e9-2881-4707-bffd-98bab02b8f41-kube-api-access-w8qzm\") pod \"payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77\" (UID: \"26d972e9-2881-4707-bffd-98bab02b8f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:21:55.898959 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.898925    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:21:55.925352 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:55.925322    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:21:56.017313 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.017240    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"]
Sep 17 14:21:56.028469 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.028115    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.029643 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.029610    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"]
Sep 17 14:21:56.031105 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:21:56.031077    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567841f6_61d0_495d_9d76_efa5f9f00db7.slice/crio-c355b0bb720ae9056dbcc395fa1a915c25686439a21d6a8a48850929e672d9fb WatchSource:0}: Error finding container c355b0bb720ae9056dbcc395fa1a915c25686439a21d6a8a48850929e672d9fb: Status 404 returned error can't find the container with id c355b0bb720ae9056dbcc395fa1a915c25686439a21d6a8a48850929e672d9fb
Sep 17 14:21:56.033156 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.033113    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8b50f2-serving-cert\""
Sep 17 14:21:56.040350 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.037638    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"]
Sep 17 14:21:56.100554 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.100519    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-metrics-certs\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.100723 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.100565    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjbq\" (UniqueName: \"kubernetes.io/projected/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-kube-api-access-nvjbq\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.100723 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.100588    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.100723 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.100670    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.175536 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.175502    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"]
Sep 17 14:21:56.178599 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:21:56.178570    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca97e45d_8be0_417d_a657_436387f6b59b.slice/crio-010f648c9757f4e8bd825b5c52197dd24bf1459cc346af034c196639e24db81f WatchSource:0}: Error finding container 010f648c9757f4e8bd825b5c52197dd24bf1459cc346af034c196639e24db81f: Status 404 returned error can't find the container with id 010f648c9757f4e8bd825b5c52197dd24bf1459cc346af034c196639e24db81f
Sep 17 14:21:56.201554 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.201529    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-metrics-certs\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.201656 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.201573    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjbq\" (UniqueName: \"kubernetes.io/projected/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-kube-api-access-nvjbq\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.201656 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.201596    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.201799 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.201777    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.202543 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.202516    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.203993 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.203967    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.204191 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.204159    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-metrics-certs\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.212131 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.212110    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"]
Sep 17 14:21:56.214669 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:21:56.214647    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d972e9_2881_4707_bffd_98bab02b8f41.slice/crio-3d9b557aca6b16bcbdace7806c99091024556ebc909db9317483dde9172b9dd7 WatchSource:0}: Error finding container 3d9b557aca6b16bcbdace7806c99091024556ebc909db9317483dde9172b9dd7: Status 404 returned error can't find the container with id 3d9b557aca6b16bcbdace7806c99091024556ebc909db9317483dde9172b9dd7
Sep 17 14:21:56.214755 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.214692    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjbq\" (UniqueName: \"kubernetes.io/projected/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-kube-api-access-nvjbq\") pod \"maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.350544 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.350472    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:56.632294 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.632256    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"]
Sep 17 14:21:56.636298 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:21:56.636261    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2dbdf8_a240_42f3_9a7c_391f9cb2a808.slice/crio-f6db82afbb78c238207ccfd230555261650d56eaa2eeac81ef634733f0336d9c WatchSource:0}: Error finding container f6db82afbb78c238207ccfd230555261650d56eaa2eeac81ef634733f0336d9c: Status 404 returned error can't find the container with id f6db82afbb78c238207ccfd230555261650d56eaa2eeac81ef634733f0336d9c
Sep 17 14:21:56.825928 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.825885    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59" event={"ID":"567841f6-61d0-495d-9d76-efa5f9f00db7","Type":"ContainerStarted","Data":"c355b0bb720ae9056dbcc395fa1a915c25686439a21d6a8a48850929e672d9fb"}
Sep 17 14:21:56.827535 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.827501    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77" event={"ID":"26d972e9-2881-4707-bffd-98bab02b8f41","Type":"ContainerStarted","Data":"4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e"}
Sep 17 14:21:56.827664 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.827554    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77" event={"ID":"26d972e9-2881-4707-bffd-98bab02b8f41","Type":"ContainerStarted","Data":"3d9b557aca6b16bcbdace7806c99091024556ebc909db9317483dde9172b9dd7"}
Sep 17 14:21:56.827740 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.827670    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:21:56.829310 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.829287    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr" event={"ID":"ca97e45d-8be0-417d-a657-436387f6b59b","Type":"ContainerStarted","Data":"bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09"}
Sep 17 14:21:56.829434 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.829328    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr" event={"ID":"ca97e45d-8be0-417d-a657-436387f6b59b","Type":"ContainerStarted","Data":"010f648c9757f4e8bd825b5c52197dd24bf1459cc346af034c196639e24db81f"}
Sep 17 14:21:56.829434 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.829378    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:21:56.830498 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.830477    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m" event={"ID":"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808","Type":"ContainerStarted","Data":"f6db82afbb78c238207ccfd230555261650d56eaa2eeac81ef634733f0336d9c"}
Sep 17 14:21:56.860466 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.860411    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77" podStartSLOduration=1.8603915149999999 podStartE2EDuration="1.860391515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:56.860238616 +0000 UTC m=+921.151352477" watchObservedRunningTime="2026-09-17 14:21:56.860391515 +0000 UTC m=+921.151505368"
Sep 17 14:21:56.882471 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.882378    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr" podStartSLOduration=1.882362134 podStartE2EDuration="1.882362134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:56.881187983 +0000 UTC m=+921.172301838" watchObservedRunningTime="2026-09-17 14:21:56.882362134 +0000 UTC m=+921.173476043"
Sep 17 14:21:56.890300 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.890273    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"]
Sep 17 14:21:56.894849 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.894827    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:21:56.900085 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.900058    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8b50f2\""
Sep 17 14:21:56.900488 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.900097    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8b50f2-dockercfg-pzhvz\""
Sep 17 14:21:56.914396 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:56.914370    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"]
Sep 17 14:21:57.008752 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.008719    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d11546-5e73-4978-8161-5af243723c30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7\" (UID: \"77d11546-5e73-4978-8161-5af243723c30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:21:57.008959 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.008767    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gs7d\" (UniqueName: \"kubernetes.io/projected/77d11546-5e73-4978-8161-5af243723c30-kube-api-access-4gs7d\") pod \"payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7\" (UID: \"77d11546-5e73-4978-8161-5af243723c30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:21:57.109990 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.109954    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d11546-5e73-4978-8161-5af243723c30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7\" (UID: \"77d11546-5e73-4978-8161-5af243723c30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:21:57.110138 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.110000    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gs7d\" (UniqueName: \"kubernetes.io/projected/77d11546-5e73-4978-8161-5af243723c30-kube-api-access-4gs7d\") pod \"payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7\" (UID: \"77d11546-5e73-4978-8161-5af243723c30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:21:57.110664 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.110643    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d11546-5e73-4978-8161-5af243723c30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7\" (UID: \"77d11546-5e73-4978-8161-5af243723c30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:21:57.120073 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.120044    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gs7d\" (UniqueName: \"kubernetes.io/projected/77d11546-5e73-4978-8161-5af243723c30-kube-api-access-4gs7d\") pod \"payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7\" (UID: \"77d11546-5e73-4978-8161-5af243723c30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:21:57.205284 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.205255    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:21:57.510937 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.510912    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"]
Sep 17 14:21:57.512494 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:21:57.512413    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d11546_5e73_4978_8161_5af243723c30.slice/crio-79e99ee2ea2525543c8308d85c5701b40fb90e5280e1de2209b134e482d0b393 WatchSource:0}: Error finding container 79e99ee2ea2525543c8308d85c5701b40fb90e5280e1de2209b134e482d0b393: Status 404 returned error can't find the container with id 79e99ee2ea2525543c8308d85c5701b40fb90e5280e1de2209b134e482d0b393
Sep 17 14:21:57.842920 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.842834    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m" event={"ID":"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808","Type":"ContainerStarted","Data":"3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9"}
Sep 17 14:21:57.843090 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.842944    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:21:57.844853 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.844822    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59" event={"ID":"567841f6-61d0-495d-9d76-efa5f9f00db7","Type":"ContainerStarted","Data":"40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27"}
Sep 17 14:21:57.844987 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.844875    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:21:57.846402 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.846375    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7" event={"ID":"77d11546-5e73-4978-8161-5af243723c30","Type":"ContainerStarted","Data":"87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29"}
Sep 17 14:21:57.846538 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.846423    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7" event={"ID":"77d11546-5e73-4978-8161-5af243723c30","Type":"ContainerStarted","Data":"79e99ee2ea2525543c8308d85c5701b40fb90e5280e1de2209b134e482d0b393"}
Sep 17 14:21:57.869536 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.869480    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m" podStartSLOduration=1.999929271 podStartE2EDuration="2.869461155s" totalImagesPullingTime="869.531884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:57.866953862 +0000 UTC m=+922.158067725" watchObservedRunningTime="2026-09-17 14:21:57.869461155 +0000 UTC m=+922.160575017"
Sep 17 14:21:57.889800 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.889750    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7" podStartSLOduration=1.889738804 podStartE2EDuration="1.889738804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:57.889585881 +0000 UTC m=+922.180699742" watchObservedRunningTime="2026-09-17 14:21:57.889738804 +0000 UTC m=+922.180852664"
Sep 17 14:21:57.942308 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:57.942257    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59" podStartSLOduration=2.792831424 podStartE2EDuration="3.942240212s" totalImagesPullingTime="1.149408788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:57.940014435 +0000 UTC m=+922.231128297" watchObservedRunningTime="2026-09-17 14:21:57.942240212 +0000 UTC m=+922.233354073"
Sep 17 14:21:58.482701 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.482661    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"]
Sep 17 14:21:58.486956 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.486932    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:21:58.490313 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.490278    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2e1b82\""
Sep 17 14:21:58.490653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.490635    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2e1b82-dockercfg-nfm8k\""
Sep 17 14:21:58.498571 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.498545    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"]
Sep 17 14:21:58.622206 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.622141    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9332b324-45d8-4249-a3b6-3cae8271ec3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq\" (UID: \"9332b324-45d8-4249-a3b6-3cae8271ec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:21:58.622206 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.622214    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn54p\" (UniqueName: \"kubernetes.io/projected/9332b324-45d8-4249-a3b6-3cae8271ec3c-kube-api-access-pn54p\") pod \"payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq\" (UID: \"9332b324-45d8-4249-a3b6-3cae8271ec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:21:58.723640 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.723601    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9332b324-45d8-4249-a3b6-3cae8271ec3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq\" (UID: \"9332b324-45d8-4249-a3b6-3cae8271ec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:21:58.723796 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.723649    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn54p\" (UniqueName: \"kubernetes.io/projected/9332b324-45d8-4249-a3b6-3cae8271ec3c-kube-api-access-pn54p\") pod \"payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq\" (UID: \"9332b324-45d8-4249-a3b6-3cae8271ec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:21:58.724416 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.724396    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9332b324-45d8-4249-a3b6-3cae8271ec3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq\" (UID: \"9332b324-45d8-4249-a3b6-3cae8271ec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:21:58.733386 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.733327    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn54p\" (UniqueName: \"kubernetes.io/projected/9332b324-45d8-4249-a3b6-3cae8271ec3c-kube-api-access-pn54p\") pod \"payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq\" (UID: \"9332b324-45d8-4249-a3b6-3cae8271ec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:21:58.801244 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.801211    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:21:58.856592 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:58.855765    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:21:59.088362 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:59.088284    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"]
Sep 17 14:21:59.091893 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:21:59.091862    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9332b324_45d8_4249_a3b6_3cae8271ec3c.slice/crio-d198f4edbe833c014e038b89c8ee16c0f1f0c4a3543792526c190caa02f97fa8 WatchSource:0}: Error finding container d198f4edbe833c014e038b89c8ee16c0f1f0c4a3543792526c190caa02f97fa8: Status 404 returned error can't find the container with id d198f4edbe833c014e038b89c8ee16c0f1f0c4a3543792526c190caa02f97fa8
Sep 17 14:21:59.863822 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:59.863772    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq" event={"ID":"9332b324-45d8-4249-a3b6-3cae8271ec3c","Type":"ContainerStarted","Data":"afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49"}
Sep 17 14:21:59.864083 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:59.863841    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq" event={"ID":"9332b324-45d8-4249-a3b6-3cae8271ec3c","Type":"ContainerStarted","Data":"d198f4edbe833c014e038b89c8ee16c0f1f0c4a3543792526c190caa02f97fa8"}
Sep 17 14:21:59.864083 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:59.864017    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:21:59.888170 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:21:59.888106    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq" podStartSLOduration=1.8880878509999999 podStartE2EDuration="1.888087851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:59.886403786 +0000 UTC m=+924.177517649" watchObservedRunningTime="2026-09-17 14:21:59.888087851 +0000 UTC m=+924.179201711"
Sep 17 14:22:03.860939 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:03.860907    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:22:03.861973 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:03.861956    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:22:07.387329 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.387292    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"]
Sep 17 14:22:07.390715 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.390696    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:22:07.399376 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.399352    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-86c464\""
Sep 17 14:22:07.400531 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.400515    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-86c464-dockercfg-h87xz\""
Sep 17 14:22:07.411894 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.411867    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"]
Sep 17 14:22:07.436246 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.436216    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"]
Sep 17 14:22:07.439559 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.439543    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:22:07.455505 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.455481    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"]
Sep 17 14:22:07.505405 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.505371    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aa2c16-067b-44d9-8d73-9086fd769a99-plugins-config-volume\") pod \"payload-processing-e2e-autores-86c464-9c9b87947-pcvkm\" (UID: \"62aa2c16-067b-44d9-8d73-9086fd769a99\") " pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:22:07.505405 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.505407    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ccs\" (UniqueName: \"kubernetes.io/projected/62aa2c16-067b-44d9-8d73-9086fd769a99-kube-api-access-n9ccs\") pod \"payload-processing-e2e-autores-86c464-9c9b87947-pcvkm\" (UID: \"62aa2c16-067b-44d9-8d73-9086fd769a99\") " pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:22:07.606048 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.606012    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24186de6-84ac-4b30-b3a6-d0953293e5c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd\" (UID: \"24186de6-84ac-4b30-b3a6-d0953293e5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:22:07.606048 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.606054    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79njn\" (UniqueName: \"kubernetes.io/projected/24186de6-84ac-4b30-b3a6-d0953293e5c1-kube-api-access-79njn\") pod \"payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd\" (UID: \"24186de6-84ac-4b30-b3a6-d0953293e5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:22:07.606332 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.606091    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aa2c16-067b-44d9-8d73-9086fd769a99-plugins-config-volume\") pod \"payload-processing-e2e-autores-86c464-9c9b87947-pcvkm\" (UID: \"62aa2c16-067b-44d9-8d73-9086fd769a99\") " pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:22:07.606332 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.606126    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ccs\" (UniqueName: \"kubernetes.io/projected/62aa2c16-067b-44d9-8d73-9086fd769a99-kube-api-access-n9ccs\") pod \"payload-processing-e2e-autores-86c464-9c9b87947-pcvkm\" (UID: \"62aa2c16-067b-44d9-8d73-9086fd769a99\") " pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:22:07.606861 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.606838    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aa2c16-067b-44d9-8d73-9086fd769a99-plugins-config-volume\") pod \"payload-processing-e2e-autores-86c464-9c9b87947-pcvkm\" (UID: \"62aa2c16-067b-44d9-8d73-9086fd769a99\") " pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:22:07.617109 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.617079    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ccs\" (UniqueName: \"kubernetes.io/projected/62aa2c16-067b-44d9-8d73-9086fd769a99-kube-api-access-n9ccs\") pod \"payload-processing-e2e-autores-86c464-9c9b87947-pcvkm\" (UID: \"62aa2c16-067b-44d9-8d73-9086fd769a99\") " pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:22:07.700729 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.700641    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:22:07.707569 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.707531    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24186de6-84ac-4b30-b3a6-d0953293e5c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd\" (UID: \"24186de6-84ac-4b30-b3a6-d0953293e5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:22:07.707710 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.707585    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79njn\" (UniqueName: \"kubernetes.io/projected/24186de6-84ac-4b30-b3a6-d0953293e5c1-kube-api-access-79njn\") pod \"payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd\" (UID: \"24186de6-84ac-4b30-b3a6-d0953293e5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:22:07.708312 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.708286    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24186de6-84ac-4b30-b3a6-d0953293e5c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd\" (UID: \"24186de6-84ac-4b30-b3a6-d0953293e5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:22:07.735739 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.735700    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79njn\" (UniqueName: \"kubernetes.io/projected/24186de6-84ac-4b30-b3a6-d0953293e5c1-kube-api-access-79njn\") pod \"payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd\" (UID: \"24186de6-84ac-4b30-b3a6-d0953293e5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:22:07.749595 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.749558    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:22:07.848272 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.848242    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:22:07.849240 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.849217    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:22:07.995718 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:07.995689    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"]
Sep 17 14:22:07.997168 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:22:07.997140    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62aa2c16_067b_44d9_8d73_9086fd769a99.slice/crio-0d5d491e189a1805335b12785396923adccef15de594ff2b667507f5054f448a WatchSource:0}: Error finding container 0d5d491e189a1805335b12785396923adccef15de594ff2b667507f5054f448a: Status 404 returned error can't find the container with id 0d5d491e189a1805335b12785396923adccef15de594ff2b667507f5054f448a
Sep 17 14:22:08.044619 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.044598    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"]
Sep 17 14:22:08.046114 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:22:08.046090    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24186de6_84ac_4b30_b3a6_d0953293e5c1.slice/crio-c1e9674a6f58bca0f600d15946b8454a2e12660c54cdc0dd1696711a58787fdb WatchSource:0}: Error finding container c1e9674a6f58bca0f600d15946b8454a2e12660c54cdc0dd1696711a58787fdb: Status 404 returned error can't find the container with id c1e9674a6f58bca0f600d15946b8454a2e12660c54cdc0dd1696711a58787fdb
Sep 17 14:22:08.534865 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.534823    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"]
Sep 17 14:22:08.538600 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.538577    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:08.543835 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.543809    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a48a4ae7\""
Sep 17 14:22:08.544547 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.544528    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a48a4ae7-dockercfg-pcd7k\""
Sep 17 14:22:08.553530 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.553504    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"]
Sep 17 14:22:08.715585 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.715553    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l4gr\" (UniqueName: \"kubernetes.io/projected/3cd3a441-3970-472a-b286-f3020133be03-kube-api-access-7l4gr\") pod \"payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c\" (UID: \"3cd3a441-3970-472a-b286-f3020133be03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:08.715755 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.715634    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd3a441-3970-472a-b286-f3020133be03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c\" (UID: \"3cd3a441-3970-472a-b286-f3020133be03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:08.817020 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.816940    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l4gr\" (UniqueName: \"kubernetes.io/projected/3cd3a441-3970-472a-b286-f3020133be03-kube-api-access-7l4gr\") pod \"payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c\" (UID: \"3cd3a441-3970-472a-b286-f3020133be03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:08.817020 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.816994    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd3a441-3970-472a-b286-f3020133be03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c\" (UID: \"3cd3a441-3970-472a-b286-f3020133be03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:08.817686 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.817663    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd3a441-3970-472a-b286-f3020133be03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c\" (UID: \"3cd3a441-3970-472a-b286-f3020133be03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:08.832023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.831996    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l4gr\" (UniqueName: \"kubernetes.io/projected/3cd3a441-3970-472a-b286-f3020133be03-kube-api-access-7l4gr\") pod \"payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c\" (UID: \"3cd3a441-3970-472a-b286-f3020133be03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:08.850027 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.850001    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:08.916018 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.915631    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd" event={"ID":"24186de6-84ac-4b30-b3a6-d0953293e5c1","Type":"ContainerStarted","Data":"576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9"}
Sep 17 14:22:08.916018 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.915707    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd" event={"ID":"24186de6-84ac-4b30-b3a6-d0953293e5c1","Type":"ContainerStarted","Data":"c1e9674a6f58bca0f600d15946b8454a2e12660c54cdc0dd1696711a58787fdb"}
Sep 17 14:22:08.916018 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.915773    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:22:08.918335 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.918306    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm" event={"ID":"62aa2c16-067b-44d9-8d73-9086fd769a99","Type":"ContainerStarted","Data":"48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b"}
Sep 17 14:22:08.918462 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.918371    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:22:08.918462 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:08.918399    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm" event={"ID":"62aa2c16-067b-44d9-8d73-9086fd769a99","Type":"ContainerStarted","Data":"0d5d491e189a1805335b12785396923adccef15de594ff2b667507f5054f448a"}
Sep 17 14:22:09.022927 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:09.022882    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd" podStartSLOduration=2.022866085 podStartE2EDuration="2.022866085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:08.95353957 +0000 UTC m=+933.244653430" watchObservedRunningTime="2026-09-17 14:22:09.022866085 +0000 UTC m=+933.313979947"
Sep 17 14:22:09.024482 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:09.024449    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm" podStartSLOduration=2.024435773 podStartE2EDuration="2.024435773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:09.020698079 +0000 UTC m=+933.311811940" watchObservedRunningTime="2026-09-17 14:22:09.024435773 +0000 UTC m=+933.315549634"
Sep 17 14:22:09.167033 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:09.167005    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"]
Sep 17 14:22:09.169484 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:22:09.169458    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd3a441_3970_472a_b286_f3020133be03.slice/crio-82f15d74d54f2739f8d50df2b7dccecd227363dd2ac5a97c558d84792082e296 WatchSource:0}: Error finding container 82f15d74d54f2739f8d50df2b7dccecd227363dd2ac5a97c558d84792082e296: Status 404 returned error can't find the container with id 82f15d74d54f2739f8d50df2b7dccecd227363dd2ac5a97c558d84792082e296
Sep 17 14:22:09.866543 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:09.866509    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:22:09.930386 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:09.930352    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c" event={"ID":"3cd3a441-3970-472a-b286-f3020133be03","Type":"ContainerStarted","Data":"c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc"}
Sep 17 14:22:09.930562 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:09.930411    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c" event={"ID":"3cd3a441-3970-472a-b286-f3020133be03","Type":"ContainerStarted","Data":"82f15d74d54f2739f8d50df2b7dccecd227363dd2ac5a97c558d84792082e296"}
Sep 17 14:22:09.955238 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:09.955155    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c" podStartSLOduration=1.955137337 podStartE2EDuration="1.955137337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:09.953370261 +0000 UTC m=+934.244484115" watchObservedRunningTime="2026-09-17 14:22:09.955137337 +0000 UTC m=+934.246251199"
Sep 17 14:22:10.872829 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:10.872797    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:22:10.935818 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:10.935786    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:19.932814 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:19.932779    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:22:19.933285 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:19.933228    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:22:20.941240 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:20.941202    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"]
Sep 17 14:22:20.944684 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:20.944660    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:20.947427 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:20.947403    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-28f011-serving-cert\""
Sep 17 14:22:20.959106 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:20.959061    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"]
Sep 17 14:22:21.031360 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.031330    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6gb\" (UniqueName: \"kubernetes.io/projected/c595490c-4a8d-4158-9380-e65cdfa61ae4-kube-api-access-2r6gb\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.031539 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.031386    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-maas-api-tls\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.031539 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.031450    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c595490c-4a8d-4158-9380-e65cdfa61ae4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.031651 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.031540    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-metrics-certs\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.132465 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.132432    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-maas-api-tls\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.132465 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.132473    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c595490c-4a8d-4158-9380-e65cdfa61ae4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.132726 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.132513    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-metrics-certs\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.132726 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.132573    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6gb\" (UniqueName: \"kubernetes.io/projected/c595490c-4a8d-4158-9380-e65cdfa61ae4-kube-api-access-2r6gb\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.133327 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.133300    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c595490c-4a8d-4158-9380-e65cdfa61ae4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.135063 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.135042    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-metrics-certs\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.135211 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.135169    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-maas-api-tls\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.148854 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.148826    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6gb\" (UniqueName: \"kubernetes.io/projected/c595490c-4a8d-4158-9380-e65cdfa61ae4-kube-api-access-2r6gb\") pod \"maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.255717 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.255628    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.534982 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.534956    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"]
Sep 17 14:22:21.537018 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:22:21.536988    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc595490c_4a8d_4158_9380_e65cdfa61ae4.slice/crio-ca50e1aeb8e54d8f173bcc1ba0e7639b0b6802c1ea49e2b24e36b9fe9a5b221d WatchSource:0}: Error finding container ca50e1aeb8e54d8f173bcc1ba0e7639b0b6802c1ea49e2b24e36b9fe9a5b221d: Status 404 returned error can't find the container with id ca50e1aeb8e54d8f173bcc1ba0e7639b0b6802c1ea49e2b24e36b9fe9a5b221d
Sep 17 14:22:21.942697 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.942672    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:21.994973 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.994940    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98" event={"ID":"c595490c-4a8d-4158-9380-e65cdfa61ae4","Type":"ContainerStarted","Data":"0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16"}
Sep 17 14:22:21.995154 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.995005    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:21.995154 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:21.995028    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98" event={"ID":"c595490c-4a8d-4158-9380-e65cdfa61ae4","Type":"ContainerStarted","Data":"ca50e1aeb8e54d8f173bcc1ba0e7639b0b6802c1ea49e2b24e36b9fe9a5b221d"}
Sep 17 14:22:22.027627 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.027576    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98" podStartSLOduration=1.677031471 podStartE2EDuration="2.027534549s" totalImagesPullingTime="350.503078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:22.027060323 +0000 UTC m=+946.318174186" watchObservedRunningTime="2026-09-17 14:22:22.027534549 +0000 UTC m=+946.318648407"
Sep 17 14:22:22.127239 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.127194    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"]
Sep 17 14:22:22.133030 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.132953    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:22:22.136915 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.136889    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-28f011-dockercfg-gr5fc\""
Sep 17 14:22:22.137459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.137439    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-28f011\""
Sep 17 14:22:22.141602 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.141533    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4nl7\" (UniqueName: \"kubernetes.io/projected/ff332d7f-b9b7-423e-95a9-34f9514a9004-kube-api-access-l4nl7\") pod \"payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7\" (UID: \"ff332d7f-b9b7-423e-95a9-34f9514a9004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:22:22.141727 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.141696    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff332d7f-b9b7-423e-95a9-34f9514a9004-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7\" (UID: \"ff332d7f-b9b7-423e-95a9-34f9514a9004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:22:22.142282 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.142263    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"]
Sep 17 14:22:22.242086 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.242056    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff332d7f-b9b7-423e-95a9-34f9514a9004-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7\" (UID: \"ff332d7f-b9b7-423e-95a9-34f9514a9004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:22:22.242262 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.242102    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nl7\" (UniqueName: \"kubernetes.io/projected/ff332d7f-b9b7-423e-95a9-34f9514a9004-kube-api-access-l4nl7\") pod \"payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7\" (UID: \"ff332d7f-b9b7-423e-95a9-34f9514a9004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:22:22.242762 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.242739    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff332d7f-b9b7-423e-95a9-34f9514a9004-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7\" (UID: \"ff332d7f-b9b7-423e-95a9-34f9514a9004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:22:22.261009 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.260967    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nl7\" (UniqueName: \"kubernetes.io/projected/ff332d7f-b9b7-423e-95a9-34f9514a9004-kube-api-access-l4nl7\") pod \"payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7\" (UID: \"ff332d7f-b9b7-423e-95a9-34f9514a9004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:22:22.458312 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.458230    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:22:22.738730 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:22.738700    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"]
Sep 17 14:22:22.740708 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:22:22.740685    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff332d7f_b9b7_423e_95a9_34f9514a9004.slice/crio-ee0794558ed32f7c47026552a305379c5600ee5e09e214204035ad569a062af1 WatchSource:0}: Error finding container ee0794558ed32f7c47026552a305379c5600ee5e09e214204035ad569a062af1: Status 404 returned error can't find the container with id ee0794558ed32f7c47026552a305379c5600ee5e09e214204035ad569a062af1
Sep 17 14:22:23.001830 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:23.001737    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7" event={"ID":"ff332d7f-b9b7-423e-95a9-34f9514a9004","Type":"ContainerStarted","Data":"1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20"}
Sep 17 14:22:23.001830 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:23.001792    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7" event={"ID":"ff332d7f-b9b7-423e-95a9-34f9514a9004","Type":"ContainerStarted","Data":"ee0794558ed32f7c47026552a305379c5600ee5e09e214204035ad569a062af1"}
Sep 17 14:22:23.002340 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:23.001863    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:22:23.022583 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:23.022533    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7" podStartSLOduration=1.022519145 podStartE2EDuration="1.022519145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:23.021571025 +0000 UTC m=+947.312684886" watchObservedRunningTime="2026-09-17 14:22:23.022519145 +0000 UTC m=+947.313633005"
Sep 17 14:22:28.007964 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:28.007935    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:22:34.009997 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:34.009963    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:22:46.492292 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:46.492260    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"]
Sep 17 14:22:46.492819 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:46.492508    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c" podUID="3cd3a441-3970-472a-b286-f3020133be03" containerName="payload-pre-processing" containerID="cri-o://c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc" gracePeriod=30
Sep 17 14:22:46.749318 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:46.749260    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:46.863711 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:46.863675    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l4gr\" (UniqueName: \"kubernetes.io/projected/3cd3a441-3970-472a-b286-f3020133be03-kube-api-access-7l4gr\") pod \"3cd3a441-3970-472a-b286-f3020133be03\" (UID: \"3cd3a441-3970-472a-b286-f3020133be03\") "
Sep 17 14:22:46.863914 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:46.863746    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd3a441-3970-472a-b286-f3020133be03-plugins-config-volume\") pod \"3cd3a441-3970-472a-b286-f3020133be03\" (UID: \"3cd3a441-3970-472a-b286-f3020133be03\") "
Sep 17 14:22:46.864455 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:46.864423    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd3a441-3970-472a-b286-f3020133be03-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3cd3a441-3970-472a-b286-f3020133be03" (UID: "3cd3a441-3970-472a-b286-f3020133be03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:22:46.866044 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:46.866017    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd3a441-3970-472a-b286-f3020133be03-kube-api-access-7l4gr" (OuterVolumeSpecName: "kube-api-access-7l4gr") pod "3cd3a441-3970-472a-b286-f3020133be03" (UID: "3cd3a441-3970-472a-b286-f3020133be03"). InnerVolumeSpecName "kube-api-access-7l4gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:22:46.964746 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:46.964713    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l4gr\" (UniqueName: \"kubernetes.io/projected/3cd3a441-3970-472a-b286-f3020133be03-kube-api-access-7l4gr\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:22:46.964746 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:46.964742    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd3a441-3970-472a-b286-f3020133be03-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:22:47.126204 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:47.126149    2537 generic.go:356] "Generic (PLEG): container finished" podID="3cd3a441-3970-472a-b286-f3020133be03" containerID="c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc" exitCode=0
Sep 17 14:22:47.126392 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:47.126216    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"
Sep 17 14:22:47.126392 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:47.126234    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c" event={"ID":"3cd3a441-3970-472a-b286-f3020133be03","Type":"ContainerDied","Data":"c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc"}
Sep 17 14:22:47.126392 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:47.126291    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c" event={"ID":"3cd3a441-3970-472a-b286-f3020133be03","Type":"ContainerDied","Data":"82f15d74d54f2739f8d50df2b7dccecd227363dd2ac5a97c558d84792082e296"}
Sep 17 14:22:47.126392 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:47.126313    2537 scope.go:117] "RemoveContainer" containerID="c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc"
Sep 17 14:22:47.135503 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:47.135483    2537 scope.go:117] "RemoveContainer" containerID="c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc"
Sep 17 14:22:47.135763 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:22:47.135744    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc\": container with ID starting with c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc not found: ID does not exist" containerID="c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc"
Sep 17 14:22:47.135825 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:47.135770    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc"} err="failed to get container status \"c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc\": rpc error: code = NotFound desc = could not find container \"c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc\": container with ID starting with c723a5d31b39703866e04757c4e221796739b4ebf734ea6ac2dacdb0535eabbc not found: ID does not exist"
Sep 17 14:22:47.159043 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:47.159019    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"]
Sep 17 14:22:47.181339 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:47.181306    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a48a4ae7-5c57d5f79d-ghx7c"]
Sep 17 14:22:48.246153 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:22:48.246110    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cd3a441-3970-472a-b286-f3020133be03" path="/var/lib/kubelet/pods/3cd3a441-3970-472a-b286-f3020133be03/volumes"
Sep 17 14:24:22.841663 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:22.841624    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"]
Sep 17 14:24:22.842245 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:22.841826    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm" podUID="62aa2c16-067b-44d9-8d73-9086fd769a99" containerName="payload-processing" containerID="cri-o://48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b" gracePeriod=30
Sep 17 14:24:22.848276 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:22.848245    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"]
Sep 17 14:24:22.848545 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:22.848495    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd" podUID="24186de6-84ac-4b30-b3a6-d0953293e5c1" containerName="payload-pre-processing" containerID="cri-o://576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9" gracePeriod=30
Sep 17 14:24:23.135121 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.135096    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:24:23.137240 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.137222    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:24:23.222524 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.222496    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aa2c16-067b-44d9-8d73-9086fd769a99-plugins-config-volume\") pod \"62aa2c16-067b-44d9-8d73-9086fd769a99\" (UID: \"62aa2c16-067b-44d9-8d73-9086fd769a99\") "
Sep 17 14:24:23.222692 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.222539    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9ccs\" (UniqueName: \"kubernetes.io/projected/62aa2c16-067b-44d9-8d73-9086fd769a99-kube-api-access-n9ccs\") pod \"62aa2c16-067b-44d9-8d73-9086fd769a99\" (UID: \"62aa2c16-067b-44d9-8d73-9086fd769a99\") "
Sep 17 14:24:23.222692 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.222575    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24186de6-84ac-4b30-b3a6-d0953293e5c1-plugins-config-volume\") pod \"24186de6-84ac-4b30-b3a6-d0953293e5c1\" (UID: \"24186de6-84ac-4b30-b3a6-d0953293e5c1\") "
Sep 17 14:24:23.222692 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.222590    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79njn\" (UniqueName: \"kubernetes.io/projected/24186de6-84ac-4b30-b3a6-d0953293e5c1-kube-api-access-79njn\") pod \"24186de6-84ac-4b30-b3a6-d0953293e5c1\" (UID: \"24186de6-84ac-4b30-b3a6-d0953293e5c1\") "
Sep 17 14:24:23.222951 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.222918    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62aa2c16-067b-44d9-8d73-9086fd769a99-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62aa2c16-067b-44d9-8d73-9086fd769a99" (UID: "62aa2c16-067b-44d9-8d73-9086fd769a99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:23.223026 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.222987    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24186de6-84ac-4b30-b3a6-d0953293e5c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24186de6-84ac-4b30-b3a6-d0953293e5c1" (UID: "24186de6-84ac-4b30-b3a6-d0953293e5c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:23.224689 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.224662    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62aa2c16-067b-44d9-8d73-9086fd769a99-kube-api-access-n9ccs" (OuterVolumeSpecName: "kube-api-access-n9ccs") pod "62aa2c16-067b-44d9-8d73-9086fd769a99" (UID: "62aa2c16-067b-44d9-8d73-9086fd769a99"). InnerVolumeSpecName "kube-api-access-n9ccs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:24:23.224805 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.224772    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24186de6-84ac-4b30-b3a6-d0953293e5c1-kube-api-access-79njn" (OuterVolumeSpecName: "kube-api-access-79njn") pod "24186de6-84ac-4b30-b3a6-d0953293e5c1" (UID: "24186de6-84ac-4b30-b3a6-d0953293e5c1"). InnerVolumeSpecName "kube-api-access-79njn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:24:23.323456 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.323419    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aa2c16-067b-44d9-8d73-9086fd769a99-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:24:23.323456 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.323451    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9ccs\" (UniqueName: \"kubernetes.io/projected/62aa2c16-067b-44d9-8d73-9086fd769a99-kube-api-access-n9ccs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:24:23.323647 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.323469    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24186de6-84ac-4b30-b3a6-d0953293e5c1-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:24:23.323647 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.323479    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79njn\" (UniqueName: \"kubernetes.io/projected/24186de6-84ac-4b30-b3a6-d0953293e5c1-kube-api-access-79njn\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:24:23.610360 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.610318    2537 generic.go:356] "Generic (PLEG): container finished" podID="24186de6-84ac-4b30-b3a6-d0953293e5c1" containerID="576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9" exitCode=0
Sep 17 14:24:23.610544 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.610377    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"
Sep 17 14:24:23.610544 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.610394    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd" event={"ID":"24186de6-84ac-4b30-b3a6-d0953293e5c1","Type":"ContainerDied","Data":"576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9"}
Sep 17 14:24:23.610544 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.610457    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd" event={"ID":"24186de6-84ac-4b30-b3a6-d0953293e5c1","Type":"ContainerDied","Data":"c1e9674a6f58bca0f600d15946b8454a2e12660c54cdc0dd1696711a58787fdb"}
Sep 17 14:24:23.610544 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.610485    2537 scope.go:117] "RemoveContainer" containerID="576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9"
Sep 17 14:24:23.611848 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.611827    2537 generic.go:356] "Generic (PLEG): container finished" podID="62aa2c16-067b-44d9-8d73-9086fd769a99" containerID="48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b" exitCode=0
Sep 17 14:24:23.611912 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.611874    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"
Sep 17 14:24:23.611962 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.611912    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm" event={"ID":"62aa2c16-067b-44d9-8d73-9086fd769a99","Type":"ContainerDied","Data":"48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b"}
Sep 17 14:24:23.611962 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.611957    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm" event={"ID":"62aa2c16-067b-44d9-8d73-9086fd769a99","Type":"ContainerDied","Data":"0d5d491e189a1805335b12785396923adccef15de594ff2b667507f5054f448a"}
Sep 17 14:24:23.621011 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.620996    2537 scope.go:117] "RemoveContainer" containerID="576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9"
Sep 17 14:24:23.621315 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:24:23.621296    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9\": container with ID starting with 576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9 not found: ID does not exist" containerID="576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9"
Sep 17 14:24:23.621386 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.621327    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9"} err="failed to get container status \"576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9\": rpc error: code = NotFound desc = could not find container \"576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9\": container with ID starting with 576e97784d26d4fc62104f9eb7325c7dfd1a269a96d39f979c58dca06dbec8c9 not found: ID does not exist"
Sep 17 14:24:23.621386 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.621348    2537 scope.go:117] "RemoveContainer" containerID="48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b"
Sep 17 14:24:23.629128 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.629112    2537 scope.go:117] "RemoveContainer" containerID="48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b"
Sep 17 14:24:23.629407 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:24:23.629392    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b\": container with ID starting with 48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b not found: ID does not exist" containerID="48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b"
Sep 17 14:24:23.629459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.629414    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b"} err="failed to get container status \"48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b\": rpc error: code = NotFound desc = could not find container \"48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b\": container with ID starting with 48f8943c550f8aacec7c99a58741a2937c3455e9008c3e851aa0d0d8ab03c04b not found: ID does not exist"
Sep 17 14:24:23.635828 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.635807    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"]
Sep 17 14:24:23.639716 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.639695    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-86c464-55dd94bb94-6pjvd"]
Sep 17 14:24:23.651190 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.651161    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"]
Sep 17 14:24:23.659128 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:23.659108    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-86c464-9c9b87947-pcvkm"]
Sep 17 14:24:24.246672 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:24.246639    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24186de6-84ac-4b30-b3a6-d0953293e5c1" path="/var/lib/kubelet/pods/24186de6-84ac-4b30-b3a6-d0953293e5c1/volumes"
Sep 17 14:24:24.247031 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:24.246986    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62aa2c16-067b-44d9-8d73-9086fd769a99" path="/var/lib/kubelet/pods/62aa2c16-067b-44d9-8d73-9086fd769a99/volumes"
Sep 17 14:24:28.512698 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.512665    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"]
Sep 17 14:24:28.513059 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.513035    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62aa2c16-067b-44d9-8d73-9086fd769a99" containerName="payload-processing"
Sep 17 14:24:28.513059 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.513048    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="62aa2c16-067b-44d9-8d73-9086fd769a99" containerName="payload-processing"
Sep 17 14:24:28.513142 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.513075    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24186de6-84ac-4b30-b3a6-d0953293e5c1" containerName="payload-pre-processing"
Sep 17 14:24:28.513142 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.513080    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="24186de6-84ac-4b30-b3a6-d0953293e5c1" containerName="payload-pre-processing"
Sep 17 14:24:28.513142 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.513093    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd3a441-3970-472a-b286-f3020133be03" containerName="payload-pre-processing"
Sep 17 14:24:28.513142 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.513099    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd3a441-3970-472a-b286-f3020133be03" containerName="payload-pre-processing"
Sep 17 14:24:28.513300 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.513169    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="24186de6-84ac-4b30-b3a6-d0953293e5c1" containerName="payload-pre-processing"
Sep 17 14:24:28.513300 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.513191    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="62aa2c16-067b-44d9-8d73-9086fd769a99" containerName="payload-processing"
Sep 17 14:24:28.513300 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.513201    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd3a441-3970-472a-b286-f3020133be03" containerName="payload-pre-processing"
Sep 17 14:24:28.517838 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.517821    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:24:28.521934 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.521914    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2f37280d-dockercfg-vhhf9\""
Sep 17 14:24:28.522103 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.521914    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2f37280d\""
Sep 17 14:24:28.534295 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.534273    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"]
Sep 17 14:24:28.571850 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.571826    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j885c\" (UniqueName: \"kubernetes.io/projected/25232764-8f68-4e0b-bd53-260f925bf7a1-kube-api-access-j885c\") pod \"payload-processing-e2e-derive-2f37280d-57dd94865-k252b\" (UID: \"25232764-8f68-4e0b-bd53-260f925bf7a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:24:28.571971 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.571858    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25232764-8f68-4e0b-bd53-260f925bf7a1-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f37280d-57dd94865-k252b\" (UID: \"25232764-8f68-4e0b-bd53-260f925bf7a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:24:28.672953 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.672891    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j885c\" (UniqueName: \"kubernetes.io/projected/25232764-8f68-4e0b-bd53-260f925bf7a1-kube-api-access-j885c\") pod \"payload-processing-e2e-derive-2f37280d-57dd94865-k252b\" (UID: \"25232764-8f68-4e0b-bd53-260f925bf7a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:24:28.672953 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.672939    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25232764-8f68-4e0b-bd53-260f925bf7a1-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f37280d-57dd94865-k252b\" (UID: \"25232764-8f68-4e0b-bd53-260f925bf7a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:24:28.673864 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.673837    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25232764-8f68-4e0b-bd53-260f925bf7a1-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f37280d-57dd94865-k252b\" (UID: \"25232764-8f68-4e0b-bd53-260f925bf7a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:24:28.684027 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.684000    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j885c\" (UniqueName: \"kubernetes.io/projected/25232764-8f68-4e0b-bd53-260f925bf7a1-kube-api-access-j885c\") pod \"payload-processing-e2e-derive-2f37280d-57dd94865-k252b\" (UID: \"25232764-8f68-4e0b-bd53-260f925bf7a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:24:28.828450 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:28.828364    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:24:29.140067 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:29.140041    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"]
Sep 17 14:24:29.142041 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:24:29.142013    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25232764_8f68_4e0b_bd53_260f925bf7a1.slice/crio-ad8e0d41d987429ce3926477a1de589eb3f925db6f20d885591016a076177500 WatchSource:0}: Error finding container ad8e0d41d987429ce3926477a1de589eb3f925db6f20d885591016a076177500: Status 404 returned error can't find the container with id ad8e0d41d987429ce3926477a1de589eb3f925db6f20d885591016a076177500
Sep 17 14:24:29.646379 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:29.646336    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b" event={"ID":"25232764-8f68-4e0b-bd53-260f925bf7a1","Type":"ContainerStarted","Data":"f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2"}
Sep 17 14:24:29.646747 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:29.646407    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:24:29.646747 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:29.646432    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b" event={"ID":"25232764-8f68-4e0b-bd53-260f925bf7a1","Type":"ContainerStarted","Data":"ad8e0d41d987429ce3926477a1de589eb3f925db6f20d885591016a076177500"}
Sep 17 14:24:29.668933 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:29.668884    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b" podStartSLOduration=1.668868581 podStartE2EDuration="1.668868581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:29.667407403 +0000 UTC m=+1073.958521264" watchObservedRunningTime="2026-09-17 14:24:29.668868581 +0000 UTC m=+1073.959982445"
Sep 17 14:24:40.653572 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:24:40.653539    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:25:06.555247 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.555214    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"]
Sep 17 14:25:06.555819 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.555498    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b" podUID="25232764-8f68-4e0b-bd53-260f925bf7a1" containerName="payload-processing" containerID="cri-o://f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2" gracePeriod=30
Sep 17 14:25:06.818206 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.818123    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:25:06.833822 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.833792    2537 generic.go:356] "Generic (PLEG): container finished" podID="25232764-8f68-4e0b-bd53-260f925bf7a1" containerID="f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2" exitCode=0
Sep 17 14:25:06.834003 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.833836    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"
Sep 17 14:25:06.834003 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.833831    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b" event={"ID":"25232764-8f68-4e0b-bd53-260f925bf7a1","Type":"ContainerDied","Data":"f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2"}
Sep 17 14:25:06.834003 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.833955    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b" event={"ID":"25232764-8f68-4e0b-bd53-260f925bf7a1","Type":"ContainerDied","Data":"ad8e0d41d987429ce3926477a1de589eb3f925db6f20d885591016a076177500"}
Sep 17 14:25:06.834003 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.833978    2537 scope.go:117] "RemoveContainer" containerID="f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2"
Sep 17 14:25:06.842406 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.842388    2537 scope.go:117] "RemoveContainer" containerID="f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2"
Sep 17 14:25:06.842735 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:25:06.842712    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2\": container with ID starting with f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2 not found: ID does not exist" containerID="f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2"
Sep 17 14:25:06.842821 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.842743    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2"} err="failed to get container status \"f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2\": rpc error: code = NotFound desc = could not find container \"f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2\": container with ID starting with f97a188ac2cebdd8cfa912b3bb1034f33cdc6cfa3dd037eefe4ccb76af3da8e2 not found: ID does not exist"
Sep 17 14:25:06.902477 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.902440    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j885c\" (UniqueName: \"kubernetes.io/projected/25232764-8f68-4e0b-bd53-260f925bf7a1-kube-api-access-j885c\") pod \"25232764-8f68-4e0b-bd53-260f925bf7a1\" (UID: \"25232764-8f68-4e0b-bd53-260f925bf7a1\") "
Sep 17 14:25:06.902641 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.902492    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25232764-8f68-4e0b-bd53-260f925bf7a1-plugins-config-volume\") pod \"25232764-8f68-4e0b-bd53-260f925bf7a1\" (UID: \"25232764-8f68-4e0b-bd53-260f925bf7a1\") "
Sep 17 14:25:06.902871 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.902850    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25232764-8f68-4e0b-bd53-260f925bf7a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25232764-8f68-4e0b-bd53-260f925bf7a1" (UID: "25232764-8f68-4e0b-bd53-260f925bf7a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:25:06.904643 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:06.904612    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25232764-8f68-4e0b-bd53-260f925bf7a1-kube-api-access-j885c" (OuterVolumeSpecName: "kube-api-access-j885c") pod "25232764-8f68-4e0b-bd53-260f925bf7a1" (UID: "25232764-8f68-4e0b-bd53-260f925bf7a1"). InnerVolumeSpecName "kube-api-access-j885c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:07.003696 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:07.003665    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j885c\" (UniqueName: \"kubernetes.io/projected/25232764-8f68-4e0b-bd53-260f925bf7a1-kube-api-access-j885c\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:25:07.003696 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:07.003692    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25232764-8f68-4e0b-bd53-260f925bf7a1-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:25:07.186893 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:07.186857    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"]
Sep 17 14:25:07.204345 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:07.204313    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37280d-57dd94865-k252b"]
Sep 17 14:25:08.245925 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:08.245893    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25232764-8f68-4e0b-bd53-260f925bf7a1" path="/var/lib/kubelet/pods/25232764-8f68-4e0b-bd53-260f925bf7a1/volumes"
Sep 17 14:25:34.527262 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.527227    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"]
Sep 17 14:25:34.527853 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.527622    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25232764-8f68-4e0b-bd53-260f925bf7a1" containerName="payload-processing"
Sep 17 14:25:34.527853 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.527634    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="25232764-8f68-4e0b-bd53-260f925bf7a1" containerName="payload-processing"
Sep 17 14:25:34.527853 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.527699    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="25232764-8f68-4e0b-bd53-260f925bf7a1" containerName="payload-processing"
Sep 17 14:25:34.530750 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.530727    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.543537 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.543518    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8b50f2\"/\"kube-root-ca.crt\""
Sep 17 14:25:34.543839 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.543822    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b50f2\"/\"default-dockercfg-pcxz4\""
Sep 17 14:25:34.543885 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.543825    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8b50f2\"/\"openshift-service-ca.crt\""
Sep 17 14:25:34.545194 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.545164    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b50f2\"/\"premium-w4-8b50f2-kserve-self-signed-certs\""
Sep 17 14:25:34.560219 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.560194    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"]
Sep 17 14:25:34.652857 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.652821    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-dshm\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.652857 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.652858    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-tmp-dir\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.653051 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.652913    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96gcm\" (UniqueName: \"kubernetes.io/projected/0b73f4fe-354f-4b09-b98d-509ba817198e-kube-api-access-96gcm\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.653051 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.652939    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-model-cache\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.653051 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.652983    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b73f4fe-354f-4b09-b98d-509ba817198e-tls-certs\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.653051 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.653009    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-home\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.753824 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.753785    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gcm\" (UniqueName: \"kubernetes.io/projected/0b73f4fe-354f-4b09-b98d-509ba817198e-kube-api-access-96gcm\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.753824 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.753830    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-model-cache\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.754063 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.753893    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b73f4fe-354f-4b09-b98d-509ba817198e-tls-certs\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.754063 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.753924    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-home\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.754063 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.753983    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-dshm\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.754063 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.754011    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-tmp-dir\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.754304 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.754283    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-model-cache\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.754372 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.754315    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-home\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.754469 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.754432    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-tmp-dir\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.756279 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.756259    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-dshm\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.756518 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.756504    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b73f4fe-354f-4b09-b98d-509ba817198e-tls-certs\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.762658 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.762632    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96gcm\" (UniqueName: \"kubernetes.io/projected/0b73f4fe-354f-4b09-b98d-509ba817198e-kube-api-access-96gcm\") pod \"premium-w4-8b50f2-kserve-56965fdb57-lvxh5\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") " pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:34.839317 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:34.839233    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:35.120501 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:35.120463    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"]
Sep 17 14:25:35.122115 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:25:35.122088    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b73f4fe_354f_4b09_b98d_509ba817198e.slice/crio-45f33d442ecb3c60e11213dabd30e8ae8bf0e84ef0ca7690cfd124c243099358 WatchSource:0}: Error finding container 45f33d442ecb3c60e11213dabd30e8ae8bf0e84ef0ca7690cfd124c243099358: Status 404 returned error can't find the container with id 45f33d442ecb3c60e11213dabd30e8ae8bf0e84ef0ca7690cfd124c243099358
Sep 17 14:25:35.974701 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:35.974670    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5" event={"ID":"0b73f4fe-354f-4b09-b98d-509ba817198e","Type":"ContainerStarted","Data":"7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641"}
Sep 17 14:25:35.975068 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:35.974709    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5" event={"ID":"0b73f4fe-354f-4b09-b98d-509ba817198e","Type":"ContainerStarted","Data":"45f33d442ecb3c60e11213dabd30e8ae8bf0e84ef0ca7690cfd124c243099358"}
Sep 17 14:25:36.003336 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:36.003284    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5" podStartSLOduration=2.003269579 podStartE2EDuration="2.003269579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:25:36.002014446 +0000 UTC m=+1140.293128307" watchObservedRunningTime="2026-09-17 14:25:36.003269579 +0000 UTC m=+1140.294383439"
Sep 17 14:25:44.840087 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:44.840049    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:44.840087 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:44.840091    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:44.853295 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:44.853263    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:25:45.028295 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:25:45.028266    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:26:36.172656 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:26:36.172630    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:26:36.174791 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:26:36.174769    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:28:19.704740 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.704705    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"]
Sep 17 14:28:19.708549 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.708523    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:28:19.713145 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.713118    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4009da-dockercfg-2668f\""
Sep 17 14:28:19.713482 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.713461    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4009da\""
Sep 17 14:28:19.731435 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.731403    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"]
Sep 17 14:28:19.750204 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.750146    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f49f99-f175-4557-b309-1a914175600b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj\" (UID: \"91f49f99-f175-4557-b309-1a914175600b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:28:19.750204 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.750201    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndf5c\" (UniqueName: \"kubernetes.io/projected/91f49f99-f175-4557-b309-1a914175600b-kube-api-access-ndf5c\") pod \"payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj\" (UID: \"91f49f99-f175-4557-b309-1a914175600b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:28:19.850884 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.850849    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f49f99-f175-4557-b309-1a914175600b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj\" (UID: \"91f49f99-f175-4557-b309-1a914175600b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:28:19.850884 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.850887    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndf5c\" (UniqueName: \"kubernetes.io/projected/91f49f99-f175-4557-b309-1a914175600b-kube-api-access-ndf5c\") pod \"payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj\" (UID: \"91f49f99-f175-4557-b309-1a914175600b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:28:19.851640 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.851622    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f49f99-f175-4557-b309-1a914175600b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj\" (UID: \"91f49f99-f175-4557-b309-1a914175600b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:28:19.865449 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.865417    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"]
Sep 17 14:28:19.868827 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.868810    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:28:19.893138 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.893103    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"]
Sep 17 14:28:19.899409 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.899386    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndf5c\" (UniqueName: \"kubernetes.io/projected/91f49f99-f175-4557-b309-1a914175600b-kube-api-access-ndf5c\") pod \"payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj\" (UID: \"91f49f99-f175-4557-b309-1a914175600b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:28:19.951376 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.951342    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fdd105-fea0-4e42-81fa-b71474e5ecde-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq\" (UID: \"41fdd105-fea0-4e42-81fa-b71474e5ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:28:19.951376 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:19.951376    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvn9f\" (UniqueName: \"kubernetes.io/projected/41fdd105-fea0-4e42-81fa-b71474e5ecde-kube-api-access-fvn9f\") pod \"payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq\" (UID: \"41fdd105-fea0-4e42-81fa-b71474e5ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:28:20.020122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.020039    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:28:20.052607 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.052575    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fdd105-fea0-4e42-81fa-b71474e5ecde-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq\" (UID: \"41fdd105-fea0-4e42-81fa-b71474e5ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:28:20.052782 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.052618    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvn9f\" (UniqueName: \"kubernetes.io/projected/41fdd105-fea0-4e42-81fa-b71474e5ecde-kube-api-access-fvn9f\") pod \"payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq\" (UID: \"41fdd105-fea0-4e42-81fa-b71474e5ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:28:20.053285 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.053264    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fdd105-fea0-4e42-81fa-b71474e5ecde-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq\" (UID: \"41fdd105-fea0-4e42-81fa-b71474e5ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:28:20.075360 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.075331    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvn9f\" (UniqueName: \"kubernetes.io/projected/41fdd105-fea0-4e42-81fa-b71474e5ecde-kube-api-access-fvn9f\") pod \"payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq\" (UID: \"41fdd105-fea0-4e42-81fa-b71474e5ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:28:20.179094 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.179058    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:28:20.313945 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.313918    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"]
Sep 17 14:28:20.316515 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:28:20.316488    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f49f99_f175_4557_b309_1a914175600b.slice/crio-e5b967a1a1c08a2724830127a0db2b81a37a19d6ea2abe10d184ad01d3f2a700 WatchSource:0}: Error finding container e5b967a1a1c08a2724830127a0db2b81a37a19d6ea2abe10d184ad01d3f2a700: Status 404 returned error can't find the container with id e5b967a1a1c08a2724830127a0db2b81a37a19d6ea2abe10d184ad01d3f2a700
Sep 17 14:28:20.460783 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.460751    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"]
Sep 17 14:28:20.463892 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:28:20.463864    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fdd105_fea0_4e42_81fa_b71474e5ecde.slice/crio-8f277ca2e83f925694b8e72d0503967e12f810cf1d3d5168459cbe4a2c6ea2a2 WatchSource:0}: Error finding container 8f277ca2e83f925694b8e72d0503967e12f810cf1d3d5168459cbe4a2c6ea2a2: Status 404 returned error can't find the container with id 8f277ca2e83f925694b8e72d0503967e12f810cf1d3d5168459cbe4a2c6ea2a2
Sep 17 14:28:20.796391 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.796357    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj" event={"ID":"91f49f99-f175-4557-b309-1a914175600b","Type":"ContainerStarted","Data":"aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f"}
Sep 17 14:28:20.796830 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.796415    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj" event={"ID":"91f49f99-f175-4557-b309-1a914175600b","Type":"ContainerStarted","Data":"e5b967a1a1c08a2724830127a0db2b81a37a19d6ea2abe10d184ad01d3f2a700"}
Sep 17 14:28:20.796830 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.796441    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:28:20.798215 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.798158    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq" event={"ID":"41fdd105-fea0-4e42-81fa-b71474e5ecde","Type":"ContainerStarted","Data":"91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772"}
Sep 17 14:28:20.798305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.798237    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq" event={"ID":"41fdd105-fea0-4e42-81fa-b71474e5ecde","Type":"ContainerStarted","Data":"8f277ca2e83f925694b8e72d0503967e12f810cf1d3d5168459cbe4a2c6ea2a2"}
Sep 17 14:28:20.798305 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.798267    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:28:20.824832 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.824775    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj" podStartSLOduration=1.824760701 podStartE2EDuration="1.824760701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:28:20.823880414 +0000 UTC m=+1305.114994276" watchObservedRunningTime="2026-09-17 14:28:20.824760701 +0000 UTC m=+1305.115874563"
Sep 17 14:28:20.853242 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:20.853171    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq" podStartSLOduration=1.853148902 podStartE2EDuration="1.853148902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:28:20.851750705 +0000 UTC m=+1305.142864566" watchObservedRunningTime="2026-09-17 14:28:20.853148902 +0000 UTC m=+1305.144262764"
Sep 17 14:28:31.805882 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:31.805850    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:28:31.806355 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:31.805923    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:28:59.271165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.271073    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"]
Sep 17 14:28:59.271721 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.271329    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq" podUID="9332b324-45d8-4249-a3b6-3cae8271ec3c" containerName="payload-pre-processing" containerID="cri-o://afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49" gracePeriod=30
Sep 17 14:28:59.534343 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.534279    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:28:59.620065 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.620030    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9332b324-45d8-4249-a3b6-3cae8271ec3c-plugins-config-volume\") pod \"9332b324-45d8-4249-a3b6-3cae8271ec3c\" (UID: \"9332b324-45d8-4249-a3b6-3cae8271ec3c\") "
Sep 17 14:28:59.620353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.620329    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9332b324-45d8-4249-a3b6-3cae8271ec3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9332b324-45d8-4249-a3b6-3cae8271ec3c" (UID: "9332b324-45d8-4249-a3b6-3cae8271ec3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:28:59.720643 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.720603    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn54p\" (UniqueName: \"kubernetes.io/projected/9332b324-45d8-4249-a3b6-3cae8271ec3c-kube-api-access-pn54p\") pod \"9332b324-45d8-4249-a3b6-3cae8271ec3c\" (UID: \"9332b324-45d8-4249-a3b6-3cae8271ec3c\") "
Sep 17 14:28:59.720820 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.720809    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9332b324-45d8-4249-a3b6-3cae8271ec3c-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:28:59.722672 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.722641    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9332b324-45d8-4249-a3b6-3cae8271ec3c-kube-api-access-pn54p" (OuterVolumeSpecName: "kube-api-access-pn54p") pod "9332b324-45d8-4249-a3b6-3cae8271ec3c" (UID: "9332b324-45d8-4249-a3b6-3cae8271ec3c"). InnerVolumeSpecName "kube-api-access-pn54p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:28:59.821336 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.821254    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn54p\" (UniqueName: \"kubernetes.io/projected/9332b324-45d8-4249-a3b6-3cae8271ec3c-kube-api-access-pn54p\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:28:59.996358 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.996328    2537 generic.go:356] "Generic (PLEG): container finished" podID="9332b324-45d8-4249-a3b6-3cae8271ec3c" containerID="afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49" exitCode=0
Sep 17 14:28:59.996526 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.996370    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"
Sep 17 14:28:59.996526 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.996415    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq" event={"ID":"9332b324-45d8-4249-a3b6-3cae8271ec3c","Type":"ContainerDied","Data":"afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49"}
Sep 17 14:28:59.996526 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.996469    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq" event={"ID":"9332b324-45d8-4249-a3b6-3cae8271ec3c","Type":"ContainerDied","Data":"d198f4edbe833c014e038b89c8ee16c0f1f0c4a3543792526c190caa02f97fa8"}
Sep 17 14:28:59.996526 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:28:59.996492    2537 scope.go:117] "RemoveContainer" containerID="afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49"
Sep 17 14:29:00.008198 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:00.008050    2537 scope.go:117] "RemoveContainer" containerID="afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49"
Sep 17 14:29:00.008841 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:29:00.008710    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49\": container with ID starting with afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49 not found: ID does not exist" containerID="afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49"
Sep 17 14:29:00.008841 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:00.008750    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49"} err="failed to get container status \"afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49\": rpc error: code = NotFound desc = could not find container \"afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49\": container with ID starting with afddc2ac5680bfba5b4e74bffbc1cfa79c84f7eaae4bf2d91372222e83c5ad49 not found: ID does not exist"
Sep 17 14:29:00.075889 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:00.075800    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"]
Sep 17 14:29:00.102500 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:00.102470    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2e1b82-844cfcbc9d-rqlhq"]
Sep 17 14:29:00.246238 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:00.246204    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9332b324-45d8-4249-a3b6-3cae8271ec3c" path="/var/lib/kubelet/pods/9332b324-45d8-4249-a3b6-3cae8271ec3c/volumes"
Sep 17 14:29:29.928957 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:29.928920    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"]
Sep 17 14:29:29.929589 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:29.929566    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9332b324-45d8-4249-a3b6-3cae8271ec3c" containerName="payload-pre-processing"
Sep 17 14:29:29.929649 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:29.929594    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="9332b324-45d8-4249-a3b6-3cae8271ec3c" containerName="payload-pre-processing"
Sep 17 14:29:29.929759 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:29.929744    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="9332b324-45d8-4249-a3b6-3cae8271ec3c" containerName="payload-pre-processing"
Sep 17 14:29:29.933321 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:29.933300    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:29:29.939722 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:29.939693    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d2d17aff\""
Sep 17 14:29:29.941216 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:29.941169    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d2d17aff-dockercfg-cskwj\""
Sep 17 14:29:29.950099 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:29.950074    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"]
Sep 17 14:29:29.986469 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:29.986444    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd367a05-4595-4f86-87a9-33d4fe79a8f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5\" (UID: \"cd367a05-4595-4f86-87a9-33d4fe79a8f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:29:29.986612 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:29.986542    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrrl\" (UniqueName: \"kubernetes.io/projected/cd367a05-4595-4f86-87a9-33d4fe79a8f6-kube-api-access-svrrl\") pod \"payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5\" (UID: \"cd367a05-4595-4f86-87a9-33d4fe79a8f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:29:30.089200 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.089132    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd367a05-4595-4f86-87a9-33d4fe79a8f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5\" (UID: \"cd367a05-4595-4f86-87a9-33d4fe79a8f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:29:30.089381 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.089239    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrrl\" (UniqueName: \"kubernetes.io/projected/cd367a05-4595-4f86-87a9-33d4fe79a8f6-kube-api-access-svrrl\") pod \"payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5\" (UID: \"cd367a05-4595-4f86-87a9-33d4fe79a8f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:29:30.090411 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.090385    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd367a05-4595-4f86-87a9-33d4fe79a8f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5\" (UID: \"cd367a05-4595-4f86-87a9-33d4fe79a8f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:29:30.101291 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.101261    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrrl\" (UniqueName: \"kubernetes.io/projected/cd367a05-4595-4f86-87a9-33d4fe79a8f6-kube-api-access-svrrl\") pod \"payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5\" (UID: \"cd367a05-4595-4f86-87a9-33d4fe79a8f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:29:30.174411 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.174379    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"]
Sep 17 14:29:30.174678 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.174650    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98" podUID="c595490c-4a8d-4158-9380-e65cdfa61ae4" containerName="maas-api" containerID="cri-o://0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16" gracePeriod=30
Sep 17 14:29:30.243916 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.243882    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:29:30.249767 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.249738    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"]
Sep 17 14:29:30.249979 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.249949    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7" podUID="ff332d7f-b9b7-423e-95a9-34f9514a9004" containerName="payload-processing" containerID="cri-o://1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20" gracePeriod=30
Sep 17 14:29:30.528505 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.528299    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:29:30.534242 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.534220    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:29:30.543077 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.543034    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"]
Sep 17 14:29:30.546621 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:29:30.546589    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd367a05_4595_4f86_87a9_33d4fe79a8f6.slice/crio-40a54d52fc73c980f9f66bf5362769ac7e04d4fca2d4c238bfdfc478cd18bcc0 WatchSource:0}: Error finding container 40a54d52fc73c980f9f66bf5362769ac7e04d4fca2d4c238bfdfc478cd18bcc0: Status 404 returned error can't find the container with id 40a54d52fc73c980f9f66bf5362769ac7e04d4fca2d4c238bfdfc478cd18bcc0
Sep 17 14:29:30.594748 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.594722    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4nl7\" (UniqueName: \"kubernetes.io/projected/ff332d7f-b9b7-423e-95a9-34f9514a9004-kube-api-access-l4nl7\") pod \"ff332d7f-b9b7-423e-95a9-34f9514a9004\" (UID: \"ff332d7f-b9b7-423e-95a9-34f9514a9004\") "
Sep 17 14:29:30.594911 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.594782    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-metrics-certs\") pod \"c595490c-4a8d-4158-9380-e65cdfa61ae4\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") "
Sep 17 14:29:30.594911 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.594826    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff332d7f-b9b7-423e-95a9-34f9514a9004-plugins-config-volume\") pod \"ff332d7f-b9b7-423e-95a9-34f9514a9004\" (UID: \"ff332d7f-b9b7-423e-95a9-34f9514a9004\") "
Sep 17 14:29:30.594911 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.594856    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-maas-api-tls\") pod \"c595490c-4a8d-4158-9380-e65cdfa61ae4\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") "
Sep 17 14:29:30.594911 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.594884    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c595490c-4a8d-4158-9380-e65cdfa61ae4-openshift-service-ca\") pod \"c595490c-4a8d-4158-9380-e65cdfa61ae4\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") "
Sep 17 14:29:30.595141 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.594965    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r6gb\" (UniqueName: \"kubernetes.io/projected/c595490c-4a8d-4158-9380-e65cdfa61ae4-kube-api-access-2r6gb\") pod \"c595490c-4a8d-4158-9380-e65cdfa61ae4\" (UID: \"c595490c-4a8d-4158-9380-e65cdfa61ae4\") "
Sep 17 14:29:30.595488 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.595439    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c595490c-4a8d-4158-9380-e65cdfa61ae4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c595490c-4a8d-4158-9380-e65cdfa61ae4" (UID: "c595490c-4a8d-4158-9380-e65cdfa61ae4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:29:30.595719 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.595683    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff332d7f-b9b7-423e-95a9-34f9514a9004-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff332d7f-b9b7-423e-95a9-34f9514a9004" (UID: "ff332d7f-b9b7-423e-95a9-34f9514a9004"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:29:30.597836 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.597772    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c595490c-4a8d-4158-9380-e65cdfa61ae4" (UID: "c595490c-4a8d-4158-9380-e65cdfa61ae4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:29:30.597989 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.597962    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff332d7f-b9b7-423e-95a9-34f9514a9004-kube-api-access-l4nl7" (OuterVolumeSpecName: "kube-api-access-l4nl7") pod "ff332d7f-b9b7-423e-95a9-34f9514a9004" (UID: "ff332d7f-b9b7-423e-95a9-34f9514a9004"). InnerVolumeSpecName "kube-api-access-l4nl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:29:30.598189 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.598160    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c595490c-4a8d-4158-9380-e65cdfa61ae4" (UID: "c595490c-4a8d-4158-9380-e65cdfa61ae4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:29:30.598282 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.598257    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c595490c-4a8d-4158-9380-e65cdfa61ae4-kube-api-access-2r6gb" (OuterVolumeSpecName: "kube-api-access-2r6gb") pod "c595490c-4a8d-4158-9380-e65cdfa61ae4" (UID: "c595490c-4a8d-4158-9380-e65cdfa61ae4"). InnerVolumeSpecName "kube-api-access-2r6gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:29:30.696395 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.696358    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r6gb\" (UniqueName: \"kubernetes.io/projected/c595490c-4a8d-4158-9380-e65cdfa61ae4-kube-api-access-2r6gb\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:29:30.696395 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.696387    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4nl7\" (UniqueName: \"kubernetes.io/projected/ff332d7f-b9b7-423e-95a9-34f9514a9004-kube-api-access-l4nl7\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:29:30.696395 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.696400    2537 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-metrics-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:29:30.696395 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.696408    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff332d7f-b9b7-423e-95a9-34f9514a9004-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:29:30.696767 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.696417    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c595490c-4a8d-4158-9380-e65cdfa61ae4-maas-api-tls\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:29:30.696767 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:30.696433    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c595490c-4a8d-4158-9380-e65cdfa61ae4-openshift-service-ca\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:29:31.155804 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.155769    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5" event={"ID":"cd367a05-4595-4f86-87a9-33d4fe79a8f6","Type":"ContainerStarted","Data":"4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db"}
Sep 17 14:29:31.156260 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.155835    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5" event={"ID":"cd367a05-4595-4f86-87a9-33d4fe79a8f6","Type":"ContainerStarted","Data":"40a54d52fc73c980f9f66bf5362769ac7e04d4fca2d4c238bfdfc478cd18bcc0"}
Sep 17 14:29:31.156260 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.155864    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:29:31.157030 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.157007    2537 generic.go:356] "Generic (PLEG): container finished" podID="ff332d7f-b9b7-423e-95a9-34f9514a9004" containerID="1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20" exitCode=0
Sep 17 14:29:31.157080 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.157044    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"
Sep 17 14:29:31.157125 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.157085    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7" event={"ID":"ff332d7f-b9b7-423e-95a9-34f9514a9004","Type":"ContainerDied","Data":"1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20"}
Sep 17 14:29:31.157200 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.157128    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7" event={"ID":"ff332d7f-b9b7-423e-95a9-34f9514a9004","Type":"ContainerDied","Data":"ee0794558ed32f7c47026552a305379c5600ee5e09e214204035ad569a062af1"}
Sep 17 14:29:31.157200 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.157151    2537 scope.go:117] "RemoveContainer" containerID="1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20"
Sep 17 14:29:31.158443 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.158422    2537 generic.go:356] "Generic (PLEG): container finished" podID="c595490c-4a8d-4158-9380-e65cdfa61ae4" containerID="0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16" exitCode=0
Sep 17 14:29:31.158547 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.158448    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98" event={"ID":"c595490c-4a8d-4158-9380-e65cdfa61ae4","Type":"ContainerDied","Data":"0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16"}
Sep 17 14:29:31.158547 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.158464    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"
Sep 17 14:29:31.158547 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.158498    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98" event={"ID":"c595490c-4a8d-4158-9380-e65cdfa61ae4","Type":"ContainerDied","Data":"ca50e1aeb8e54d8f173bcc1ba0e7639b0b6802c1ea49e2b24e36b9fe9a5b221d"}
Sep 17 14:29:31.167200 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.167032    2537 scope.go:117] "RemoveContainer" containerID="1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20"
Sep 17 14:29:31.167364 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:29:31.167336    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20\": container with ID starting with 1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20 not found: ID does not exist" containerID="1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20"
Sep 17 14:29:31.167423 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.167369    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20"} err="failed to get container status \"1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20\": rpc error: code = NotFound desc = could not find container \"1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20\": container with ID starting with 1ba34b3cacdb235380857627cd657a2b06dc6e0331c094052383aad9c30aed20 not found: ID does not exist"
Sep 17 14:29:31.167423 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.167391    2537 scope.go:117] "RemoveContainer" containerID="0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16"
Sep 17 14:29:31.177739 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.177718    2537 scope.go:117] "RemoveContainer" containerID="0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16"
Sep 17 14:29:31.178122 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:29:31.178103    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16\": container with ID starting with 0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16 not found: ID does not exist" containerID="0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16"
Sep 17 14:29:31.178255 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.178128    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16"} err="failed to get container status \"0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16\": rpc error: code = NotFound desc = could not find container \"0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16\": container with ID starting with 0ecaf7dc416092a5f5613b5c8179675a4d2c390f63a6ba23b181c3b75a9d3c16 not found: ID does not exist"
Sep 17 14:29:31.187959 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.183114    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5" podStartSLOduration=2.183096173 podStartE2EDuration="2.183096173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:29:31.179352496 +0000 UTC m=+1375.470466357" watchObservedRunningTime="2026-09-17 14:29:31.183096173 +0000 UTC m=+1375.474210041"
Sep 17 14:29:31.206226 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.206197    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"]
Sep 17 14:29:31.211082 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.211058    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28f011-6bcd7f4887-x7pf7"]
Sep 17 14:29:31.225982 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.225957    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"]
Sep 17 14:29:31.233250 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:31.233227    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-28f011-5f94d8c7fc-mqf98"]
Sep 17 14:29:32.246789 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:32.246756    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c595490c-4a8d-4158-9380-e65cdfa61ae4" path="/var/lib/kubelet/pods/c595490c-4a8d-4158-9380-e65cdfa61ae4/volumes"
Sep 17 14:29:32.247150 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:32.247121    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff332d7f-b9b7-423e-95a9-34f9514a9004" path="/var/lib/kubelet/pods/ff332d7f-b9b7-423e-95a9-34f9514a9004/volumes"
Sep 17 14:29:42.167824 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:42.167793    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:29:50.537490 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.537457    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"]
Sep 17 14:29:50.537937 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.537830    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c595490c-4a8d-4158-9380-e65cdfa61ae4" containerName="maas-api"
Sep 17 14:29:50.537937 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.537841    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="c595490c-4a8d-4158-9380-e65cdfa61ae4" containerName="maas-api"
Sep 17 14:29:50.537937 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.537860    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff332d7f-b9b7-423e-95a9-34f9514a9004" containerName="payload-processing"
Sep 17 14:29:50.537937 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.537866    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff332d7f-b9b7-423e-95a9-34f9514a9004" containerName="payload-processing"
Sep 17 14:29:50.537937 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.537932    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff332d7f-b9b7-423e-95a9-34f9514a9004" containerName="payload-processing"
Sep 17 14:29:50.537937 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.537940    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="c595490c-4a8d-4158-9380-e65cdfa61ae4" containerName="maas-api"
Sep 17 14:29:50.542419 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.542403    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:50.545425 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.545392    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-3da4be-serving-cert\""
Sep 17 14:29:50.553779 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.553755    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"]
Sep 17 14:29:50.584441 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.584409    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:50.584724 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.584685    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:50.584898 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.584877    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797e0e0c-1e96-47a9-b5dd-505facd731b4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:50.585086 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.585062    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45s5l\" (UniqueName: \"kubernetes.io/projected/797e0e0c-1e96-47a9-b5dd-505facd731b4-kube-api-access-45s5l\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:50.685923 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.685887    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45s5l\" (UniqueName: \"kubernetes.io/projected/797e0e0c-1e96-47a9-b5dd-505facd731b4-kube-api-access-45s5l\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:50.686085 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.685952    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:50.686085 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.685984    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:50.686085 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.686006    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797e0e0c-1e96-47a9-b5dd-505facd731b4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:50.686352 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:29:50.686137    2537 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-serving-cert: secret "maas-api-e2e-shared-a-w3-3da4be-serving-cert" not found
Sep 17 14:29:50.686352 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:29:50.686232    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-maas-api-tls podName:797e0e0c-1e96-47a9-b5dd-505facd731b4 nodeName:}" failed. No retries permitted until 2026-09-17 14:29:51.186209778 +0000 UTC m=+1395.477323617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-maas-api-tls") pod "maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj" (UID: "797e0e0c-1e96-47a9-b5dd-505facd731b4") : secret "maas-api-e2e-shared-a-w3-3da4be-serving-cert" not found
Sep 17 14:29:50.686696 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.686678    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797e0e0c-1e96-47a9-b5dd-505facd731b4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:50.688500 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.688479    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:50.694775 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:50.694749    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45s5l\" (UniqueName: \"kubernetes.io/projected/797e0e0c-1e96-47a9-b5dd-505facd731b4-kube-api-access-45s5l\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:51.189421 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.189385    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:51.191880 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.191854    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:51.456057 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.455980    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:51.485353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.485316    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"]
Sep 17 14:29:51.489082 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.489059    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:29:51.492287 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.492263    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-3da4be\""
Sep 17 14:29:51.492477 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.492463    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-3da4be-dockercfg-rmp9z\""
Sep 17 14:29:51.503827 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.503752    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"]
Sep 17 14:29:51.592483 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.592453    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2426d0ee-79d8-4edc-94cd-1128816cdbd4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj\" (UID: \"2426d0ee-79d8-4edc-94cd-1128816cdbd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:29:51.592845 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.592560    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmntz\" (UniqueName: \"kubernetes.io/projected/2426d0ee-79d8-4edc-94cd-1128816cdbd4-kube-api-access-jmntz\") pod \"payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj\" (UID: \"2426d0ee-79d8-4edc-94cd-1128816cdbd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:29:51.693946 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.693912    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2426d0ee-79d8-4edc-94cd-1128816cdbd4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj\" (UID: \"2426d0ee-79d8-4edc-94cd-1128816cdbd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:29:51.694141 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.694026    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmntz\" (UniqueName: \"kubernetes.io/projected/2426d0ee-79d8-4edc-94cd-1128816cdbd4-kube-api-access-jmntz\") pod \"payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj\" (UID: \"2426d0ee-79d8-4edc-94cd-1128816cdbd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:29:51.694665 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.694636    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2426d0ee-79d8-4edc-94cd-1128816cdbd4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj\" (UID: \"2426d0ee-79d8-4edc-94cd-1128816cdbd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:29:51.704612 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.704581    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmntz\" (UniqueName: \"kubernetes.io/projected/2426d0ee-79d8-4edc-94cd-1128816cdbd4-kube-api-access-jmntz\") pod \"payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj\" (UID: \"2426d0ee-79d8-4edc-94cd-1128816cdbd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:29:51.770276 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.770252    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"]
Sep 17 14:29:51.772127 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:29:51.772098    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797e0e0c_1e96_47a9_b5dd_505facd731b4.slice/crio-40b52c72dcec2ee661a71f65abbb616eb51e3653cfbc66caaa82307b6cdfdebf WatchSource:0}: Error finding container 40b52c72dcec2ee661a71f65abbb616eb51e3653cfbc66caaa82307b6cdfdebf: Status 404 returned error can't find the container with id 40b52c72dcec2ee661a71f65abbb616eb51e3653cfbc66caaa82307b6cdfdebf
Sep 17 14:29:51.774000 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.773982    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:29:51.806467 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:51.806441    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:29:52.092631 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:52.092609    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"]
Sep 17 14:29:52.094784 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:29:52.094743    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2426d0ee_79d8_4edc_94cd_1128816cdbd4.slice/crio-3ee40739bc6557ba122ea601469a8efd866c49e3e1be3b4ddfe58c6d14d781c2 WatchSource:0}: Error finding container 3ee40739bc6557ba122ea601469a8efd866c49e3e1be3b4ddfe58c6d14d781c2: Status 404 returned error can't find the container with id 3ee40739bc6557ba122ea601469a8efd866c49e3e1be3b4ddfe58c6d14d781c2
Sep 17 14:29:52.267441 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:52.267404    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj" event={"ID":"2426d0ee-79d8-4edc-94cd-1128816cdbd4","Type":"ContainerStarted","Data":"37620d838f4808152669402a8067a63b8420d3314eeeee716d1e680511c958ac"}
Sep 17 14:29:52.267646 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:52.267466    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj" event={"ID":"2426d0ee-79d8-4edc-94cd-1128816cdbd4","Type":"ContainerStarted","Data":"3ee40739bc6557ba122ea601469a8efd866c49e3e1be3b4ddfe58c6d14d781c2"}
Sep 17 14:29:52.267646 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:52.267578    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:29:52.269087 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:52.269065    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj" event={"ID":"797e0e0c-1e96-47a9-b5dd-505facd731b4","Type":"ContainerStarted","Data":"70fb7b349d21785ac8a8c467ae99012f622d3bfddbd108551a2e8f9016fbfe6d"}
Sep 17 14:29:52.269227 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:52.269107    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj" event={"ID":"797e0e0c-1e96-47a9-b5dd-505facd731b4","Type":"ContainerStarted","Data":"40b52c72dcec2ee661a71f65abbb616eb51e3653cfbc66caaa82307b6cdfdebf"}
Sep 17 14:29:52.269227 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:52.269136    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:29:52.289042 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:52.288998    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj" podStartSLOduration=1.288986671 podStartE2EDuration="1.288986671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:29:52.286958057 +0000 UTC m=+1396.578071919" watchObservedRunningTime="2026-09-17 14:29:52.288986671 +0000 UTC m=+1396.580100532"
Sep 17 14:29:52.310770 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:52.310724    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj" podStartSLOduration=1.939522454 podStartE2EDuration="2.310710924s" totalImagesPullingTime="371.18847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:29:52.309444571 +0000 UTC m=+1396.600558444" watchObservedRunningTime="2026-09-17 14:29:52.310710924 +0000 UTC m=+1396.601824784"
Sep 17 14:29:58.280466 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:29:58.280438    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:30:00.265731 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:00.265697    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp"]
Sep 17 14:30:00.269401 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:00.269380    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp"
Sep 17 14:30:00.273472 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:00.273451    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:30:00.279845 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:00.279823    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp"
Sep 17 14:30:00.285752 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:00.285728    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp"]
Sep 17 14:30:00.570722 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:30:00.570691    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb157323_75ca_4878_913a_51cc95537f8a.slice/crio-cd5f65a0af6b7b9bfbb1b1424f8c74f0d96f8ff41885df0d68d8db16d2da61c3 WatchSource:0}: Error finding container cd5f65a0af6b7b9bfbb1b1424f8c74f0d96f8ff41885df0d68d8db16d2da61c3: Status 404 returned error can't find the container with id cd5f65a0af6b7b9bfbb1b1424f8c74f0d96f8ff41885df0d68d8db16d2da61c3
Sep 17 14:30:00.572520 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:00.572489    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp"]
Sep 17 14:30:01.315134 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:01.315097    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp" event={"ID":"bb157323-75ca-4878-913a-51cc95537f8a","Type":"ContainerStarted","Data":"cd5f65a0af6b7b9bfbb1b1424f8c74f0d96f8ff41885df0d68d8db16d2da61c3"}
Sep 17 14:30:03.277995 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:03.277918    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:30:03.333807 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:03.333770    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp" event={"ID":"bb157323-75ca-4878-913a-51cc95537f8a","Type":"ContainerStarted","Data":"b471e33034d951399f35246e8db238d4106c17424d86113f11b3b5756cfc588f"}
Sep 17 14:30:03.376502 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:03.376430    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp" podStartSLOduration=1.04257223 podStartE2EDuration="3.376410168s" totalImagesPullingTime="2.333837938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:03.373509688 +0000 UTC m=+1407.664623550" watchObservedRunningTime="2026-09-17 14:30:03.376410168 +0000 UTC m=+1407.667524030"
Sep 17 14:30:04.341544 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:04.341510    2537 generic.go:356] "Generic (PLEG): container finished" podID="bb157323-75ca-4878-913a-51cc95537f8a" containerID="b471e33034d951399f35246e8db238d4106c17424d86113f11b3b5756cfc588f" exitCode=0
Sep 17 14:30:04.341922 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:04.341602    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp" event={"ID":"bb157323-75ca-4878-913a-51cc95537f8a","Type":"ContainerDied","Data":"b471e33034d951399f35246e8db238d4106c17424d86113f11b3b5756cfc588f"}
Sep 17 14:30:05.476082 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:05.476051    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp"
Sep 17 14:30:06.352293 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:06.352263    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp"
Sep 17 14:30:06.352496 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:06.352301    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp" event={"ID":"bb157323-75ca-4878-913a-51cc95537f8a","Type":"ContainerDied","Data":"cd5f65a0af6b7b9bfbb1b1424f8c74f0d96f8ff41885df0d68d8db16d2da61c3"}
Sep 17 14:30:06.352496 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:06.352357    2537 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd5f65a0af6b7b9bfbb1b1424f8c74f0d96f8ff41885df0d68d8db16d2da61c3"
Sep 17 14:30:21.650302 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.650267    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"]
Sep 17 14:30:21.650894 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.650879    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb157323-75ca-4878-913a-51cc95537f8a" containerName="cleanup"
Sep 17 14:30:21.650973 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.650900    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb157323-75ca-4878-913a-51cc95537f8a" containerName="cleanup"
Sep 17 14:30:21.651035 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.651027    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb157323-75ca-4878-913a-51cc95537f8a" containerName="cleanup"
Sep 17 14:30:21.657777 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.657747    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:30:21.662758 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.662731    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-b48b7c-dockercfg-5sq7f\""
Sep 17 14:30:21.662931 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.662788    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b48b7c\""
Sep 17 14:30:21.663825 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.663806    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"]
Sep 17 14:30:21.667419 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.667399    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:30:21.671334 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.671313    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"]
Sep 17 14:30:21.689296 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.689267    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"]
Sep 17 14:30:21.767663 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.767631    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvcl7\" (UniqueName: \"kubernetes.io/projected/13a1932a-e1dd-4444-9b05-7dd008bd2d33-kube-api-access-bvcl7\") pod \"payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp\" (UID: \"13a1932a-e1dd-4444-9b05-7dd008bd2d33\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:30:21.767861 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.767683    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4\" (UID: \"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:30:21.767861 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.767790    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a1932a-e1dd-4444-9b05-7dd008bd2d33-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp\" (UID: \"13a1932a-e1dd-4444-9b05-7dd008bd2d33\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:30:21.767861 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.767837    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchgx\" (UniqueName: \"kubernetes.io/projected/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-kube-api-access-lchgx\") pod \"payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4\" (UID: \"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:30:21.869016 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.868977    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchgx\" (UniqueName: \"kubernetes.io/projected/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-kube-api-access-lchgx\") pod \"payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4\" (UID: \"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:30:21.869215 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.869068    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvcl7\" (UniqueName: \"kubernetes.io/projected/13a1932a-e1dd-4444-9b05-7dd008bd2d33-kube-api-access-bvcl7\") pod \"payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp\" (UID: \"13a1932a-e1dd-4444-9b05-7dd008bd2d33\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:30:21.869215 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.869110    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4\" (UID: \"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:30:21.869390 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.869228    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a1932a-e1dd-4444-9b05-7dd008bd2d33-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp\" (UID: \"13a1932a-e1dd-4444-9b05-7dd008bd2d33\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:30:21.869984 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.869960    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4\" (UID: \"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:30:21.870082 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.869965    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a1932a-e1dd-4444-9b05-7dd008bd2d33-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp\" (UID: \"13a1932a-e1dd-4444-9b05-7dd008bd2d33\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:30:21.887111 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.887078    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchgx\" (UniqueName: \"kubernetes.io/projected/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-kube-api-access-lchgx\") pod \"payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4\" (UID: \"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:30:21.887249 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.887226    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvcl7\" (UniqueName: \"kubernetes.io/projected/13a1932a-e1dd-4444-9b05-7dd008bd2d33-kube-api-access-bvcl7\") pod \"payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp\" (UID: \"13a1932a-e1dd-4444-9b05-7dd008bd2d33\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:30:21.969941 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.969856    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:30:21.979072 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:21.979045    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:30:22.285414 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:30:22.285385    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0b1b85_0f3a_447f_a2ea_5dd4077974d6.slice/crio-3660d736ed16ef9aa78e6d10d4872f562675304d18a19a3c6e274f447f473602 WatchSource:0}: Error finding container 3660d736ed16ef9aa78e6d10d4872f562675304d18a19a3c6e274f447f473602: Status 404 returned error can't find the container with id 3660d736ed16ef9aa78e6d10d4872f562675304d18a19a3c6e274f447f473602
Sep 17 14:30:22.285972 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:22.285924    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"]
Sep 17 14:30:22.297769 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:22.297746    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"]
Sep 17 14:30:22.302360 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:30:22.302339    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a1932a_e1dd_4444_9b05_7dd008bd2d33.slice/crio-5122ef5ecb83cf12f57951f78235753f54e3396905aa663ccaf356af4abe929a WatchSource:0}: Error finding container 5122ef5ecb83cf12f57951f78235753f54e3396905aa663ccaf356af4abe929a: Status 404 returned error can't find the container with id 5122ef5ecb83cf12f57951f78235753f54e3396905aa663ccaf356af4abe929a
Sep 17 14:30:22.439166 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:22.439127    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp" event={"ID":"13a1932a-e1dd-4444-9b05-7dd008bd2d33","Type":"ContainerStarted","Data":"acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f"}
Sep 17 14:30:22.439363 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:22.439213    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:30:22.439363 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:22.439238    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp" event={"ID":"13a1932a-e1dd-4444-9b05-7dd008bd2d33","Type":"ContainerStarted","Data":"5122ef5ecb83cf12f57951f78235753f54e3396905aa663ccaf356af4abe929a"}
Sep 17 14:30:22.441089 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:22.441061    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4" event={"ID":"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6","Type":"ContainerStarted","Data":"451f88d81485eff4112bfd4322327542b40e17c255d8c0a0960be4178a89756b"}
Sep 17 14:30:22.441244 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:22.441116    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:30:22.441244 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:22.441142    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4" event={"ID":"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6","Type":"ContainerStarted","Data":"3660d736ed16ef9aa78e6d10d4872f562675304d18a19a3c6e274f447f473602"}
Sep 17 14:30:22.471023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:22.470971    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp" podStartSLOduration=1.470925929 podStartE2EDuration="1.470925929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:22.469643421 +0000 UTC m=+1426.760757285" watchObservedRunningTime="2026-09-17 14:30:22.470925929 +0000 UTC m=+1426.762039789"
Sep 17 14:30:22.493704 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:22.493649    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4" podStartSLOduration=1.493635238 podStartE2EDuration="1.493635238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:22.49287436 +0000 UTC m=+1426.783988221" watchObservedRunningTime="2026-09-17 14:30:22.493635238 +0000 UTC m=+1426.784749098"
Sep 17 14:30:33.448339 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:33.448260    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:30:33.448896 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:30:33.448341    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:31:28.432330 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.432291    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"]
Sep 17 14:31:28.432885 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.432584    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5" podUID="cd367a05-4595-4f86-87a9-33d4fe79a8f6" containerName="payload-pre-processing" containerID="cri-o://4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db" gracePeriod=30
Sep 17 14:31:28.698552 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.698522    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:31:28.780146 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.780116    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svrrl\" (UniqueName: \"kubernetes.io/projected/cd367a05-4595-4f86-87a9-33d4fe79a8f6-kube-api-access-svrrl\") pod \"cd367a05-4595-4f86-87a9-33d4fe79a8f6\" (UID: \"cd367a05-4595-4f86-87a9-33d4fe79a8f6\") "
Sep 17 14:31:28.780146 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.780150    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd367a05-4595-4f86-87a9-33d4fe79a8f6-plugins-config-volume\") pod \"cd367a05-4595-4f86-87a9-33d4fe79a8f6\" (UID: \"cd367a05-4595-4f86-87a9-33d4fe79a8f6\") "
Sep 17 14:31:28.780612 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.780569    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd367a05-4595-4f86-87a9-33d4fe79a8f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd367a05-4595-4f86-87a9-33d4fe79a8f6" (UID: "cd367a05-4595-4f86-87a9-33d4fe79a8f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:28.782501 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.782463    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd367a05-4595-4f86-87a9-33d4fe79a8f6-kube-api-access-svrrl" (OuterVolumeSpecName: "kube-api-access-svrrl") pod "cd367a05-4595-4f86-87a9-33d4fe79a8f6" (UID: "cd367a05-4595-4f86-87a9-33d4fe79a8f6"). InnerVolumeSpecName "kube-api-access-svrrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:28.799765 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.799731    2537 generic.go:356] "Generic (PLEG): container finished" podID="cd367a05-4595-4f86-87a9-33d4fe79a8f6" containerID="4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db" exitCode=0
Sep 17 14:31:28.799870 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.799782    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"
Sep 17 14:31:28.799870 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.799814    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5" event={"ID":"cd367a05-4595-4f86-87a9-33d4fe79a8f6","Type":"ContainerDied","Data":"4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db"}
Sep 17 14:31:28.799942 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.799873    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5" event={"ID":"cd367a05-4595-4f86-87a9-33d4fe79a8f6","Type":"ContainerDied","Data":"40a54d52fc73c980f9f66bf5362769ac7e04d4fca2d4c238bfdfc478cd18bcc0"}
Sep 17 14:31:28.799942 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.799898    2537 scope.go:117] "RemoveContainer" containerID="4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db"
Sep 17 14:31:28.811758 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.811738    2537 scope.go:117] "RemoveContainer" containerID="4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db"
Sep 17 14:31:28.812117 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:31:28.812096    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db\": container with ID starting with 4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db not found: ID does not exist" containerID="4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db"
Sep 17 14:31:28.812195 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.812128    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db"} err="failed to get container status \"4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db\": rpc error: code = NotFound desc = could not find container \"4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db\": container with ID starting with 4218af18acf155bddc89d7ea9d36f51fab4b14ae7a254030cf78acfcc65b36db not found: ID does not exist"
Sep 17 14:31:28.830496 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.830472    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"]
Sep 17 14:31:28.837901 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.837876    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2d17aff-79db8d6c9d-gcsx5"]
Sep 17 14:31:28.880805 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.880775    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svrrl\" (UniqueName: \"kubernetes.io/projected/cd367a05-4595-4f86-87a9-33d4fe79a8f6-kube-api-access-svrrl\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:31:28.880805 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:28.880800    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd367a05-4595-4f86-87a9-33d4fe79a8f6-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:31:30.245841 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:30.245807    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd367a05-4595-4f86-87a9-33d4fe79a8f6" path="/var/lib/kubelet/pods/cd367a05-4595-4f86-87a9-33d4fe79a8f6/volumes"
Sep 17 14:31:36.202885 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:36.202855    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:31:36.205697 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:36.205676    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:31:46.419895 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.419863    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"]
Sep 17 14:31:46.420378 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.420323    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd367a05-4595-4f86-87a9-33d4fe79a8f6" containerName="payload-pre-processing"
Sep 17 14:31:46.420378 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.420338    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd367a05-4595-4f86-87a9-33d4fe79a8f6" containerName="payload-pre-processing"
Sep 17 14:31:46.420451 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.420411    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd367a05-4595-4f86-87a9-33d4fe79a8f6" containerName="payload-pre-processing"
Sep 17 14:31:46.423405 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.423388    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.426752 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.426730    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b50f2\"/\"distinct-w4-8b50f2-kserve-self-signed-certs\""
Sep 17 14:31:46.433834 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.433811    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"]
Sep 17 14:31:46.537134 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.537100    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-model-cache\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.537134 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.537139    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-home\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.537421 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.537165    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tls-certs\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.537421 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.537262    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54l7\" (UniqueName: \"kubernetes.io/projected/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-kube-api-access-z54l7\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.537421 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.537309    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-dshm\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.537421 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.537352    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tmp-dir\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.638044 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.638003    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-model-cache\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.638044 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.638050    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-home\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.638386 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.638066    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tls-certs\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.638386 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.638097    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54l7\" (UniqueName: \"kubernetes.io/projected/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-kube-api-access-z54l7\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.638386 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.638129    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-dshm\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.638386 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.638146    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tmp-dir\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.638601 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.638526    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-home\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.638601 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.638580    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-model-cache\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.638601 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.638595    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tmp-dir\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.640409 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.640387    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-dshm\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.640694 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.640677    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tls-certs\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.647342 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.647322    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54l7\" (UniqueName: \"kubernetes.io/projected/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-kube-api-access-z54l7\") pod \"distinct-w4-8b50f2-kserve-69bb696c76-gbt27\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:46.732887 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:46.732805    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:47.008859 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:47.008833    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"]
Sep 17 14:31:47.011313 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:31:47.011282    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bf58dc3_dcc6_40b5_8391_11cb03ee25e8.slice/crio-94ab7c7bc4b826d53ed5699dc96335bdc6b9e457aa23392266fde095cafb3372 WatchSource:0}: Error finding container 94ab7c7bc4b826d53ed5699dc96335bdc6b9e457aa23392266fde095cafb3372: Status 404 returned error can't find the container with id 94ab7c7bc4b826d53ed5699dc96335bdc6b9e457aa23392266fde095cafb3372
Sep 17 14:31:47.914920 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:47.914889    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27" event={"ID":"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8","Type":"ContainerStarted","Data":"775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e"}
Sep 17 14:31:47.915333 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:47.914929    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27" event={"ID":"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8","Type":"ContainerStarted","Data":"94ab7c7bc4b826d53ed5699dc96335bdc6b9e457aa23392266fde095cafb3372"}
Sep 17 14:31:47.937053 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:47.936997    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27" podStartSLOduration=1.936981632 podStartE2EDuration="1.936981632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:31:47.935547784 +0000 UTC m=+1512.226661743" watchObservedRunningTime="2026-09-17 14:31:47.936981632 +0000 UTC m=+1512.228095495"
Sep 17 14:31:56.733627 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:56.733586    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:56.734039 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:56.733639    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:56.746606 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:56.746572    2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:31:56.970023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:31:56.969992    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:34:43.217676 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.217643    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"]
Sep 17 14:34:43.221010 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.220991    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.223941 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.223921    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0829c1-serving-cert\""
Sep 17 14:34:43.231957 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.231935    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"]
Sep 17 14:34:43.307332 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.307299    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-maas-api-tls\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.307496 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.307350    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-openshift-service-ca\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.307496 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.307370    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-metrics-certs\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.307496 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.307443    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l64b9\" (UniqueName: \"kubernetes.io/projected/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-kube-api-access-l64b9\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.408360 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.408325    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-maas-api-tls\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.408553 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.408381    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-openshift-service-ca\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.408553 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.408471    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-metrics-certs\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.408553 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.408503    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64b9\" (UniqueName: \"kubernetes.io/projected/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-kube-api-access-l64b9\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.409130 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.409107    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-openshift-service-ca\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.410852 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.410829    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-metrics-certs\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.410937 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.410869    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-maas-api-tls\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.417219 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.417197    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l64b9\" (UniqueName: \"kubernetes.io/projected/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-kube-api-access-l64b9\") pod \"maas-api-e2e-api-b-0829c1-67485dd675-twrfl\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.533451 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.533379    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:43.809414 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.809386    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"]
Sep 17 14:34:43.811480 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:34:43.811452    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a5a5c8_0c81_4b59_b65a_5a0c072bfaea.slice/crio-d854de460350d5a5ab14fca4f6a9297204d8cf5557000b95c2b1d7499f8c0d9f WatchSource:0}: Error finding container d854de460350d5a5ab14fca4f6a9297204d8cf5557000b95c2b1d7499f8c0d9f: Status 404 returned error can't find the container with id d854de460350d5a5ab14fca4f6a9297204d8cf5557000b95c2b1d7499f8c0d9f
Sep 17 14:34:43.820913 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.820880    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl" event={"ID":"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea","Type":"ContainerStarted","Data":"d854de460350d5a5ab14fca4f6a9297204d8cf5557000b95c2b1d7499f8c0d9f"}
Sep 17 14:34:43.862459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.862433    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"]
Sep 17 14:34:43.866946 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.866929    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:34:43.869810 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.869794    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0829c1\""
Sep 17 14:34:43.869980 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.869960    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0829c1-dockercfg-hfdf4\""
Sep 17 14:34:43.877891 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:43.877863    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"]
Sep 17 14:34:44.014524 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.014481    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z\" (UID: \"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:34:44.014725 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.014598    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn84t\" (UniqueName: \"kubernetes.io/projected/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-kube-api-access-gn84t\") pod \"payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z\" (UID: \"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:34:44.032223 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.032139    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"]
Sep 17 14:34:44.032481 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.032441    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59" podUID="567841f6-61d0-495d-9d76-efa5f9f00db7" containerName="maas-api" containerID="cri-o://40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27" gracePeriod=30
Sep 17 14:34:44.063801 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.063578    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp"]
Sep 17 14:34:44.066342 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.066314    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-080801-29827590-lqvrp"]
Sep 17 14:34:44.104396 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.104363    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"]
Sep 17 14:34:44.104626 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.104601    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77" podUID="26d972e9-2881-4707-bffd-98bab02b8f41" containerName="payload-processing" containerID="cri-o://4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e" gracePeriod=30
Sep 17 14:34:44.115742 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.115213    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z\" (UID: \"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:34:44.115742 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.115336    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn84t\" (UniqueName: \"kubernetes.io/projected/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-kube-api-access-gn84t\") pod \"payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z\" (UID: \"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:34:44.116624 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.116450    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z\" (UID: \"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:34:44.129742 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.129710    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn84t\" (UniqueName: \"kubernetes.io/projected/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-kube-api-access-gn84t\") pod \"payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z\" (UID: \"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:34:44.141960 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.141929    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"]
Sep 17 14:34:44.142167 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.142131    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr" podUID="ca97e45d-8be0-417d-a657-436387f6b59b" containerName="payload-pre-processing" containerID="cri-o://bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09" gracePeriod=30
Sep 17 14:34:44.178231 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.178213    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:34:44.250500 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.250101    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb157323-75ca-4878-913a-51cc95537f8a" path="/var/lib/kubelet/pods/bb157323-75ca-4878-913a-51cc95537f8a/volumes"
Sep 17 14:34:44.406517 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.406493    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:34:44.465972 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.465941    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"]
Sep 17 14:34:44.467447 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:34:44.467417    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fdaa68_ad7c_48ce_bb3e_7597a282ccd7.slice/crio-f6f501d4dc7802e517c191c0b5ba8c90e479866458980972349f4b4909b65b0d WatchSource:0}: Error finding container f6f501d4dc7802e517c191c0b5ba8c90e479866458980972349f4b4909b65b0d: Status 404 returned error can't find the container with id f6f501d4dc7802e517c191c0b5ba8c90e479866458980972349f4b4909b65b0d
Sep 17 14:34:44.525126 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.525088    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-metrics-certs\") pod \"567841f6-61d0-495d-9d76-efa5f9f00db7\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") "
Sep 17 14:34:44.525346 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.525306    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgs2p\" (UniqueName: \"kubernetes.io/projected/567841f6-61d0-495d-9d76-efa5f9f00db7-kube-api-access-pgs2p\") pod \"567841f6-61d0-495d-9d76-efa5f9f00db7\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") "
Sep 17 14:34:44.525424 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.525348    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567841f6-61d0-495d-9d76-efa5f9f00db7-openshift-service-ca\") pod \"567841f6-61d0-495d-9d76-efa5f9f00db7\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") "
Sep 17 14:34:44.525424 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.525376    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-maas-api-tls\") pod \"567841f6-61d0-495d-9d76-efa5f9f00db7\" (UID: \"567841f6-61d0-495d-9d76-efa5f9f00db7\") "
Sep 17 14:34:44.525727 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.525698    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567841f6-61d0-495d-9d76-efa5f9f00db7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "567841f6-61d0-495d-9d76-efa5f9f00db7" (UID: "567841f6-61d0-495d-9d76-efa5f9f00db7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:34:44.527852 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.527813    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "567841f6-61d0-495d-9d76-efa5f9f00db7" (UID: "567841f6-61d0-495d-9d76-efa5f9f00db7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:34:44.527852 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.527846    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "567841f6-61d0-495d-9d76-efa5f9f00db7" (UID: "567841f6-61d0-495d-9d76-efa5f9f00db7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:34:44.528122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.528101    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567841f6-61d0-495d-9d76-efa5f9f00db7-kube-api-access-pgs2p" (OuterVolumeSpecName: "kube-api-access-pgs2p") pod "567841f6-61d0-495d-9d76-efa5f9f00db7" (UID: "567841f6-61d0-495d-9d76-efa5f9f00db7"). InnerVolumeSpecName "kube-api-access-pgs2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:34:44.571824 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.571792    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:34:44.599956 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.599921    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:34:44.627267 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.627207    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgs2p\" (UniqueName: \"kubernetes.io/projected/567841f6-61d0-495d-9d76-efa5f9f00db7-kube-api-access-pgs2p\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.627267 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.627238    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567841f6-61d0-495d-9d76-efa5f9f00db7-openshift-service-ca\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.627267 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.627252    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-maas-api-tls\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.627267 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.627264    2537 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567841f6-61d0-495d-9d76-efa5f9f00db7-metrics-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.728013 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.727920    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8qzm\" (UniqueName: \"kubernetes.io/projected/26d972e9-2881-4707-bffd-98bab02b8f41-kube-api-access-w8qzm\") pod \"26d972e9-2881-4707-bffd-98bab02b8f41\" (UID: \"26d972e9-2881-4707-bffd-98bab02b8f41\") "
Sep 17 14:34:44.728013 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.727971    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d972e9-2881-4707-bffd-98bab02b8f41-plugins-config-volume\") pod \"26d972e9-2881-4707-bffd-98bab02b8f41\" (UID: \"26d972e9-2881-4707-bffd-98bab02b8f41\") "
Sep 17 14:34:44.728264 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.728039    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6pk5\" (UniqueName: \"kubernetes.io/projected/ca97e45d-8be0-417d-a657-436387f6b59b-kube-api-access-m6pk5\") pod \"ca97e45d-8be0-417d-a657-436387f6b59b\" (UID: \"ca97e45d-8be0-417d-a657-436387f6b59b\") "
Sep 17 14:34:44.728264 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.728087    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca97e45d-8be0-417d-a657-436387f6b59b-plugins-config-volume\") pod \"ca97e45d-8be0-417d-a657-436387f6b59b\" (UID: \"ca97e45d-8be0-417d-a657-436387f6b59b\") "
Sep 17 14:34:44.728384 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.728363    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d972e9-2881-4707-bffd-98bab02b8f41-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26d972e9-2881-4707-bffd-98bab02b8f41" (UID: "26d972e9-2881-4707-bffd-98bab02b8f41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:34:44.728492 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.728468    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d972e9-2881-4707-bffd-98bab02b8f41-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.728621 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.728532    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca97e45d-8be0-417d-a657-436387f6b59b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca97e45d-8be0-417d-a657-436387f6b59b" (UID: "ca97e45d-8be0-417d-a657-436387f6b59b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:34:44.730583 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.730557    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca97e45d-8be0-417d-a657-436387f6b59b-kube-api-access-m6pk5" (OuterVolumeSpecName: "kube-api-access-m6pk5") pod "ca97e45d-8be0-417d-a657-436387f6b59b" (UID: "ca97e45d-8be0-417d-a657-436387f6b59b"). InnerVolumeSpecName "kube-api-access-m6pk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:34:44.730880 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.730858    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d972e9-2881-4707-bffd-98bab02b8f41-kube-api-access-w8qzm" (OuterVolumeSpecName: "kube-api-access-w8qzm") pod "26d972e9-2881-4707-bffd-98bab02b8f41" (UID: "26d972e9-2881-4707-bffd-98bab02b8f41"). InnerVolumeSpecName "kube-api-access-w8qzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:34:44.827962 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.827920    2537 generic.go:356] "Generic (PLEG): container finished" podID="567841f6-61d0-495d-9d76-efa5f9f00db7" containerID="40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27" exitCode=0
Sep 17 14:34:44.828157 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.827965    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"
Sep 17 14:34:44.828157 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.827997    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59" event={"ID":"567841f6-61d0-495d-9d76-efa5f9f00db7","Type":"ContainerDied","Data":"40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27"}
Sep 17 14:34:44.828157 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.828052    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59" event={"ID":"567841f6-61d0-495d-9d76-efa5f9f00db7","Type":"ContainerDied","Data":"c355b0bb720ae9056dbcc395fa1a915c25686439a21d6a8a48850929e672d9fb"}
Sep 17 14:34:44.828157 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.828086    2537 scope.go:117] "RemoveContainer" containerID="40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27"
Sep 17 14:34:44.829015 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.828995    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca97e45d-8be0-417d-a657-436387f6b59b-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.829149 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.829017    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8qzm\" (UniqueName: \"kubernetes.io/projected/26d972e9-2881-4707-bffd-98bab02b8f41-kube-api-access-w8qzm\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.829149 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.829032    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6pk5\" (UniqueName: \"kubernetes.io/projected/ca97e45d-8be0-417d-a657-436387f6b59b-kube-api-access-m6pk5\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.830108 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.830078    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z" event={"ID":"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7","Type":"ContainerStarted","Data":"c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6"}
Sep 17 14:34:44.830261 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.830135    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z" event={"ID":"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7","Type":"ContainerStarted","Data":"f6f501d4dc7802e517c191c0b5ba8c90e479866458980972349f4b4909b65b0d"}
Sep 17 14:34:44.830261 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.830165    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:34:44.831542 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.831517    2537 generic.go:356] "Generic (PLEG): container finished" podID="26d972e9-2881-4707-bffd-98bab02b8f41" containerID="4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e" exitCode=0
Sep 17 14:34:44.831656 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.831571    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"
Sep 17 14:34:44.831656 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.831609    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77" event={"ID":"26d972e9-2881-4707-bffd-98bab02b8f41","Type":"ContainerDied","Data":"4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e"}
Sep 17 14:34:44.831656 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.831651    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77" event={"ID":"26d972e9-2881-4707-bffd-98bab02b8f41","Type":"ContainerDied","Data":"3d9b557aca6b16bcbdace7806c99091024556ebc909db9317483dde9172b9dd7"}
Sep 17 14:34:44.833461 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.833437    2537 generic.go:356] "Generic (PLEG): container finished" podID="ca97e45d-8be0-417d-a657-436387f6b59b" containerID="bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09" exitCode=0
Sep 17 14:34:44.833563 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.833522    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr" event={"ID":"ca97e45d-8be0-417d-a657-436387f6b59b","Type":"ContainerDied","Data":"bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09"}
Sep 17 14:34:44.833563 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.833555    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"
Sep 17 14:34:44.833831 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.833561    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr" event={"ID":"ca97e45d-8be0-417d-a657-436387f6b59b","Type":"ContainerDied","Data":"010f648c9757f4e8bd825b5c52197dd24bf1459cc346af034c196639e24db81f"}
Sep 17 14:34:44.835323 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.835253    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl" event={"ID":"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea","Type":"ContainerStarted","Data":"608e976489ff52b5b7ccbfa00c00337cb714ce15b3d1a78d048f45dc5d35e52d"}
Sep 17 14:34:44.835450 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.835377    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:44.839667 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.839641    2537 scope.go:117] "RemoveContainer" containerID="40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27"
Sep 17 14:34:44.840621 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:34:44.840485    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27\": container with ID starting with 40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27 not found: ID does not exist" containerID="40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27"
Sep 17 14:34:44.840621 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.840522    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27"} err="failed to get container status \"40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27\": rpc error: code = NotFound desc = could not find container \"40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27\": container with ID starting with 40c6aa5b5b687151a4e139b534f484e969f628ce8d660ef7ad0010c28f03ff27 not found: ID does not exist"
Sep 17 14:34:44.840621 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.840544    2537 scope.go:117] "RemoveContainer" containerID="4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e"
Sep 17 14:34:44.851472 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.851454    2537 scope.go:117] "RemoveContainer" containerID="4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e"
Sep 17 14:34:44.851783 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:34:44.851759    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e\": container with ID starting with 4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e not found: ID does not exist" containerID="4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e"
Sep 17 14:34:44.851837 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.851793    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e"} err="failed to get container status \"4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e\": rpc error: code = NotFound desc = could not find container \"4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e\": container with ID starting with 4de38cabbf6838e1364643f5e69325dfcca50a54745d92b03b9610953b5ed41e not found: ID does not exist"
Sep 17 14:34:44.851837 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.851816    2537 scope.go:117] "RemoveContainer" containerID="bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09"
Sep 17 14:34:44.859704 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.858654    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z" podStartSLOduration=1.858639441 podStartE2EDuration="1.858639441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:34:44.856349685 +0000 UTC m=+1689.147463558" watchObservedRunningTime="2026-09-17 14:34:44.858639441 +0000 UTC m=+1689.149753303"
Sep 17 14:34:44.864256 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.864232    2537 scope.go:117] "RemoveContainer" containerID="bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09"
Sep 17 14:34:44.864631 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:34:44.864604    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09\": container with ID starting with bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09 not found: ID does not exist" containerID="bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09"
Sep 17 14:34:44.864713 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.864640    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09"} err="failed to get container status \"bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09\": rpc error: code = NotFound desc = could not find container \"bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09\": container with ID starting with bde195494df0905b846af15ce5a7d2dee5e346c3b4a14036851aca31ae62ff09 not found: ID does not exist"
Sep 17 14:34:44.874797 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.874774    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"]
Sep 17 14:34:44.876949 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.876928    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-080801-6df746bf4-lxfmr"]
Sep 17 14:34:44.896367 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.896324    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl" podStartSLOduration=1.535908156 podStartE2EDuration="1.896308055s" totalImagesPullingTime="360.399899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:34:44.893527125 +0000 UTC m=+1689.184640981" watchObservedRunningTime="2026-09-17 14:34:44.896308055 +0000 UTC m=+1689.187421909"
Sep 17 14:34:44.915904 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.915872    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"]
Sep 17 14:34:44.922486 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.922458    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-080801-6d94794bc6-7fv59"]
Sep 17 14:34:44.943195 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.943158    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"]
Sep 17 14:34:44.946336 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:44.946314    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-080801-84db66fd65-6kw77"]
Sep 17 14:34:46.246546 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:46.246514    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26d972e9-2881-4707-bffd-98bab02b8f41" path="/var/lib/kubelet/pods/26d972e9-2881-4707-bffd-98bab02b8f41/volumes"
Sep 17 14:34:46.246913 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:46.246876    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="567841f6-61d0-495d-9d76-efa5f9f00db7" path="/var/lib/kubelet/pods/567841f6-61d0-495d-9d76-efa5f9f00db7/volumes"
Sep 17 14:34:46.247208 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:46.247193    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca97e45d-8be0-417d-a657-436387f6b59b" path="/var/lib/kubelet/pods/ca97e45d-8be0-417d-a657-436387f6b59b/volumes"
Sep 17 14:34:50.850693 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:50.850663    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:34:55.848341 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:34:55.848301    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:35:32.918733 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:32.918699    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"]
Sep 17 14:35:32.919352 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:32.918945    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl" podUID="30a5a5c8-0c81-4b59-b65a-5a0c072bfaea" containerName="maas-api" containerID="cri-o://608e976489ff52b5b7ccbfa00c00337cb714ce15b3d1a78d048f45dc5d35e52d" gracePeriod=30
Sep 17 14:35:32.979883 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:32.979850    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"]
Sep 17 14:35:32.980111 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:32.980069    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z" podUID="90fdaa68-ad7c-48ce-bb3e-7597a282ccd7" containerName="payload-processing" containerID="cri-o://c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6" gracePeriod=30
Sep 17 14:35:33.094308 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.094271    2537 generic.go:356] "Generic (PLEG): container finished" podID="30a5a5c8-0c81-4b59-b65a-5a0c072bfaea" containerID="608e976489ff52b5b7ccbfa00c00337cb714ce15b3d1a78d048f45dc5d35e52d" exitCode=0
Sep 17 14:35:33.094469 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.094339    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl" event={"ID":"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea","Type":"ContainerDied","Data":"608e976489ff52b5b7ccbfa00c00337cb714ce15b3d1a78d048f45dc5d35e52d"}
Sep 17 14:35:33.243577 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.243546    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:35:33.281831 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.281798    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:35:33.389758 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.389726    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-plugins-config-volume\") pod \"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7\" (UID: \"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7\") "
Sep 17 14:35:33.389758 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.389763    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l64b9\" (UniqueName: \"kubernetes.io/projected/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-kube-api-access-l64b9\") pod \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") "
Sep 17 14:35:33.390002 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.389781    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn84t\" (UniqueName: \"kubernetes.io/projected/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-kube-api-access-gn84t\") pod \"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7\" (UID: \"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7\") "
Sep 17 14:35:33.390002 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.389805    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-maas-api-tls\") pod \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") "
Sep 17 14:35:33.390002 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.389857    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-openshift-service-ca\") pod \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") "
Sep 17 14:35:33.390002 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.389897    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-metrics-certs\") pod \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\" (UID: \"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea\") "
Sep 17 14:35:33.390333 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.390302    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30a5a5c8-0c81-4b59-b65a-5a0c072bfaea" (UID: "30a5a5c8-0c81-4b59-b65a-5a0c072bfaea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:35:33.390423 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.390392    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90fdaa68-ad7c-48ce-bb3e-7597a282ccd7" (UID: "90fdaa68-ad7c-48ce-bb3e-7597a282ccd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:35:33.391993 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.391963    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "30a5a5c8-0c81-4b59-b65a-5a0c072bfaea" (UID: "30a5a5c8-0c81-4b59-b65a-5a0c072bfaea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:35:33.391993 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.391982    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-kube-api-access-gn84t" (OuterVolumeSpecName: "kube-api-access-gn84t") pod "90fdaa68-ad7c-48ce-bb3e-7597a282ccd7" (UID: "90fdaa68-ad7c-48ce-bb3e-7597a282ccd7"). InnerVolumeSpecName "kube-api-access-gn84t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:35:33.392148 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.391973    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30a5a5c8-0c81-4b59-b65a-5a0c072bfaea" (UID: "30a5a5c8-0c81-4b59-b65a-5a0c072bfaea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:35:33.392148 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.392035    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-kube-api-access-l64b9" (OuterVolumeSpecName: "kube-api-access-l64b9") pod "30a5a5c8-0c81-4b59-b65a-5a0c072bfaea" (UID: "30a5a5c8-0c81-4b59-b65a-5a0c072bfaea"). InnerVolumeSpecName "kube-api-access-l64b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:35:33.491563 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.491481    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:35:33.491563 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.491509    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l64b9\" (UniqueName: \"kubernetes.io/projected/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-kube-api-access-l64b9\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:35:33.491563 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.491519    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn84t\" (UniqueName: \"kubernetes.io/projected/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7-kube-api-access-gn84t\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:35:33.491563 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.491529    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-maas-api-tls\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:35:33.491563 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.491538    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-openshift-service-ca\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:35:33.491563 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:33.491547    2537 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea-metrics-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:35:34.101135 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.101096    2537 generic.go:356] "Generic (PLEG): container finished" podID="90fdaa68-ad7c-48ce-bb3e-7597a282ccd7" containerID="c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6" exitCode=0
Sep 17 14:35:34.101135 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.101140    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"
Sep 17 14:35:34.101647 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.101195    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z" event={"ID":"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7","Type":"ContainerDied","Data":"c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6"}
Sep 17 14:35:34.101647 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.101251    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z" event={"ID":"90fdaa68-ad7c-48ce-bb3e-7597a282ccd7","Type":"ContainerDied","Data":"f6f501d4dc7802e517c191c0b5ba8c90e479866458980972349f4b4909b65b0d"}
Sep 17 14:35:34.101647 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.101280    2537 scope.go:117] "RemoveContainer" containerID="c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6"
Sep 17 14:35:34.102803 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.102771    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl" event={"ID":"30a5a5c8-0c81-4b59-b65a-5a0c072bfaea","Type":"ContainerDied","Data":"d854de460350d5a5ab14fca4f6a9297204d8cf5557000b95c2b1d7499f8c0d9f"}
Sep 17 14:35:34.102960 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.102802    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"
Sep 17 14:35:34.112429 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.112377    2537 scope.go:117] "RemoveContainer" containerID="c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6"
Sep 17 14:35:34.112820 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:35:34.112790    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6\": container with ID starting with c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6 not found: ID does not exist" containerID="c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6"
Sep 17 14:35:34.112930 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.112831    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6"} err="failed to get container status \"c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6\": rpc error: code = NotFound desc = could not find container \"c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6\": container with ID starting with c328c2c1c71735e013f5c66cb0b1657f82d2d29a7569e4642c748d9fb401eca6 not found: ID does not exist"
Sep 17 14:35:34.112930 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.112855    2537 scope.go:117] "RemoveContainer" containerID="608e976489ff52b5b7ccbfa00c00337cb714ce15b3d1a78d048f45dc5d35e52d"
Sep 17 14:35:34.125040 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.125015    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"]
Sep 17 14:35:34.130323 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.130294    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0829c1-5cf68c645f-nqk5z"]
Sep 17 14:35:34.142311 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.142280    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"]
Sep 17 14:35:34.144099 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.144074    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0829c1-67485dd675-twrfl"]
Sep 17 14:35:34.246235 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.246153    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a5a5c8-0c81-4b59-b65a-5a0c072bfaea" path="/var/lib/kubelet/pods/30a5a5c8-0c81-4b59-b65a-5a0c072bfaea/volumes"
Sep 17 14:35:34.246556 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:34.246542    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90fdaa68-ad7c-48ce-bb3e-7597a282ccd7" path="/var/lib/kubelet/pods/90fdaa68-ad7c-48ce-bb3e-7597a282ccd7/volumes"
Sep 17 14:35:50.675566 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:50.675532    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"]
Sep 17 14:35:50.676093 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:50.675784    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj" podUID="91f49f99-f175-4557-b309-1a914175600b" containerName="payload-processing" containerID="cri-o://aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f" gracePeriod=30
Sep 17 14:35:50.683956 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:50.683929    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"]
Sep 17 14:35:50.684217 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:50.684132    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq" podUID="41fdd105-fea0-4e42-81fa-b71474e5ecde" containerName="payload-pre-processing" containerID="cri-o://91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772" gracePeriod=30
Sep 17 14:35:50.958076 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:50.958049    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:35:50.975452 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:50.975418    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:35:51.058714 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.058679    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvn9f\" (UniqueName: \"kubernetes.io/projected/41fdd105-fea0-4e42-81fa-b71474e5ecde-kube-api-access-fvn9f\") pod \"41fdd105-fea0-4e42-81fa-b71474e5ecde\" (UID: \"41fdd105-fea0-4e42-81fa-b71474e5ecde\") "
Sep 17 14:35:51.058899 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.058765    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f49f99-f175-4557-b309-1a914175600b-plugins-config-volume\") pod \"91f49f99-f175-4557-b309-1a914175600b\" (UID: \"91f49f99-f175-4557-b309-1a914175600b\") "
Sep 17 14:35:51.058899 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.058814    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fdd105-fea0-4e42-81fa-b71474e5ecde-plugins-config-volume\") pod \"41fdd105-fea0-4e42-81fa-b71474e5ecde\" (UID: \"41fdd105-fea0-4e42-81fa-b71474e5ecde\") "
Sep 17 14:35:51.058899 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.058838    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndf5c\" (UniqueName: \"kubernetes.io/projected/91f49f99-f175-4557-b309-1a914175600b-kube-api-access-ndf5c\") pod \"91f49f99-f175-4557-b309-1a914175600b\" (UID: \"91f49f99-f175-4557-b309-1a914175600b\") "
Sep 17 14:35:51.059218 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.059156    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fdd105-fea0-4e42-81fa-b71474e5ecde-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41fdd105-fea0-4e42-81fa-b71474e5ecde" (UID: "41fdd105-fea0-4e42-81fa-b71474e5ecde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:35:51.059340 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.059251    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f49f99-f175-4557-b309-1a914175600b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91f49f99-f175-4557-b309-1a914175600b" (UID: "91f49f99-f175-4557-b309-1a914175600b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:35:51.060939 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.060916    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fdd105-fea0-4e42-81fa-b71474e5ecde-kube-api-access-fvn9f" (OuterVolumeSpecName: "kube-api-access-fvn9f") pod "41fdd105-fea0-4e42-81fa-b71474e5ecde" (UID: "41fdd105-fea0-4e42-81fa-b71474e5ecde"). InnerVolumeSpecName "kube-api-access-fvn9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:35:51.061039 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.060960    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f49f99-f175-4557-b309-1a914175600b-kube-api-access-ndf5c" (OuterVolumeSpecName: "kube-api-access-ndf5c") pod "91f49f99-f175-4557-b309-1a914175600b" (UID: "91f49f99-f175-4557-b309-1a914175600b"). InnerVolumeSpecName "kube-api-access-ndf5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:35:51.159799 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.159743    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fdd105-fea0-4e42-81fa-b71474e5ecde-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:35:51.159799 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.159791    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndf5c\" (UniqueName: \"kubernetes.io/projected/91f49f99-f175-4557-b309-1a914175600b-kube-api-access-ndf5c\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:35:51.159799 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.159804    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvn9f\" (UniqueName: \"kubernetes.io/projected/41fdd105-fea0-4e42-81fa-b71474e5ecde-kube-api-access-fvn9f\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:35:51.160047 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.159817    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f49f99-f175-4557-b309-1a914175600b-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:35:51.191677 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.191592    2537 generic.go:356] "Generic (PLEG): container finished" podID="91f49f99-f175-4557-b309-1a914175600b" containerID="aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f" exitCode=0
Sep 17 14:35:51.191677 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.191638    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"
Sep 17 14:35:51.191884 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.191679    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj" event={"ID":"91f49f99-f175-4557-b309-1a914175600b","Type":"ContainerDied","Data":"aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f"}
Sep 17 14:35:51.191884 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.191736    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj" event={"ID":"91f49f99-f175-4557-b309-1a914175600b","Type":"ContainerDied","Data":"e5b967a1a1c08a2724830127a0db2b81a37a19d6ea2abe10d184ad01d3f2a700"}
Sep 17 14:35:51.191884 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.191763    2537 scope.go:117] "RemoveContainer" containerID="aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f"
Sep 17 14:35:51.193154 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.193132    2537 generic.go:356] "Generic (PLEG): container finished" podID="41fdd105-fea0-4e42-81fa-b71474e5ecde" containerID="91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772" exitCode=0
Sep 17 14:35:51.193300 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.193167    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"
Sep 17 14:35:51.193300 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.193165    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq" event={"ID":"41fdd105-fea0-4e42-81fa-b71474e5ecde","Type":"ContainerDied","Data":"91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772"}
Sep 17 14:35:51.193300 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.193245    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq" event={"ID":"41fdd105-fea0-4e42-81fa-b71474e5ecde","Type":"ContainerDied","Data":"8f277ca2e83f925694b8e72d0503967e12f810cf1d3d5168459cbe4a2c6ea2a2"}
Sep 17 14:35:51.202227 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.202206    2537 scope.go:117] "RemoveContainer" containerID="aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f"
Sep 17 14:35:51.202534 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:35:51.202510    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f\": container with ID starting with aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f not found: ID does not exist" containerID="aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f"
Sep 17 14:35:51.202608 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.202544    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f"} err="failed to get container status \"aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f\": rpc error: code = NotFound desc = could not find container \"aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f\": container with ID starting with aedbd653306aa411ec0c3966ebb616ebade3dba10d81e961b32d0c5de2d9661f not found: ID does not exist"
Sep 17 14:35:51.202608 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.202568    2537 scope.go:117] "RemoveContainer" containerID="91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772"
Sep 17 14:35:51.210889 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.210871    2537 scope.go:117] "RemoveContainer" containerID="91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772"
Sep 17 14:35:51.211153 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:35:51.211132    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772\": container with ID starting with 91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772 not found: ID does not exist" containerID="91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772"
Sep 17 14:35:51.211274 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.211158    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772"} err="failed to get container status \"91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772\": rpc error: code = NotFound desc = could not find container \"91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772\": container with ID starting with 91a137d31cd6f5e8160a920589c38657d3a4f0be1878496958fa5fb524ad8772 not found: ID does not exist"
Sep 17 14:35:51.216130 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.216108    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"]
Sep 17 14:35:51.220473 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.220452    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4009da-6bcf4c54b5-gfhtj"]
Sep 17 14:35:51.232651 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.232629    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"]
Sep 17 14:35:51.236591 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:51.236569    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4009da-7665d67c74-cstkq"]
Sep 17 14:35:52.246016 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:52.245986    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41fdd105-fea0-4e42-81fa-b71474e5ecde" path="/var/lib/kubelet/pods/41fdd105-fea0-4e42-81fa-b71474e5ecde/volumes"
Sep 17 14:35:52.246453 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:35:52.246437    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91f49f99-f175-4557-b309-1a914175600b" path="/var/lib/kubelet/pods/91f49f99-f175-4557-b309-1a914175600b/volumes"
Sep 17 14:36:31.780362 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:31.780277    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"]
Sep 17 14:36:31.792701 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:31.792668    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"]
Sep 17 14:36:36.231108 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:36.231079    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:36:36.234792 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:36.234770    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:36:36.301708 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:36.301681    2537 scope.go:117] "RemoveContainer" containerID="b471e33034d951399f35246e8db238d4106c17424d86113f11b3b5756cfc588f"
Sep 17 14:36:42.448596 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.448561    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"]
Sep 17 14:36:42.449265 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.449144    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m" podUID="cb2dbdf8-a240-42f3-9a7c-391f9cb2a808" containerName="maas-api" containerID="cri-o://3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9" gracePeriod=30
Sep 17 14:36:42.509144 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.509103    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"]
Sep 17 14:36:42.509434 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.509388    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7" podUID="77d11546-5e73-4978-8161-5af243723c30" containerName="payload-pre-processing" containerID="cri-o://87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29" gracePeriod=30
Sep 17 14:36:42.791959 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.791924    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:36:42.818345 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.818310    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gs7d\" (UniqueName: \"kubernetes.io/projected/77d11546-5e73-4978-8161-5af243723c30-kube-api-access-4gs7d\") pod \"77d11546-5e73-4978-8161-5af243723c30\" (UID: \"77d11546-5e73-4978-8161-5af243723c30\") "
Sep 17 14:36:42.818546 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.818528    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d11546-5e73-4978-8161-5af243723c30-plugins-config-volume\") pod \"77d11546-5e73-4978-8161-5af243723c30\" (UID: \"77d11546-5e73-4978-8161-5af243723c30\") "
Sep 17 14:36:42.818918 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.818888    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d11546-5e73-4978-8161-5af243723c30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77d11546-5e73-4978-8161-5af243723c30" (UID: "77d11546-5e73-4978-8161-5af243723c30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:36:42.820465 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.820431    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d11546-5e73-4978-8161-5af243723c30-kube-api-access-4gs7d" (OuterVolumeSpecName: "kube-api-access-4gs7d") pod "77d11546-5e73-4978-8161-5af243723c30" (UID: "77d11546-5e73-4978-8161-5af243723c30"). InnerVolumeSpecName "kube-api-access-4gs7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:42.915705 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.915681    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:36:42.919694 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.919675    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gs7d\" (UniqueName: \"kubernetes.io/projected/77d11546-5e73-4978-8161-5af243723c30-kube-api-access-4gs7d\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:42.919752 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:42.919697    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d11546-5e73-4978-8161-5af243723c30-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:43.020232 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.020110    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvjbq\" (UniqueName: \"kubernetes.io/projected/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-kube-api-access-nvjbq\") pod \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") "
Sep 17 14:36:43.020232 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.020189    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-maas-api-tls\") pod \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") "
Sep 17 14:36:43.020232 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.020213    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-openshift-service-ca\") pod \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") "
Sep 17 14:36:43.020527 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.020278    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-metrics-certs\") pod \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\" (UID: \"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808\") "
Sep 17 14:36:43.020649 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.020618    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb2dbdf8-a240-42f3-9a7c-391f9cb2a808" (UID: "cb2dbdf8-a240-42f3-9a7c-391f9cb2a808"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:36:43.022189 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.022156    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-kube-api-access-nvjbq" (OuterVolumeSpecName: "kube-api-access-nvjbq") pod "cb2dbdf8-a240-42f3-9a7c-391f9cb2a808" (UID: "cb2dbdf8-a240-42f3-9a7c-391f9cb2a808"). InnerVolumeSpecName "kube-api-access-nvjbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:43.022334 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.022245    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb2dbdf8-a240-42f3-9a7c-391f9cb2a808" (UID: "cb2dbdf8-a240-42f3-9a7c-391f9cb2a808"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:36:43.022334 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.022293    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cb2dbdf8-a240-42f3-9a7c-391f9cb2a808" (UID: "cb2dbdf8-a240-42f3-9a7c-391f9cb2a808"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:36:43.121355 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.121317    2537 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-metrics-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:43.121355 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.121348    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvjbq\" (UniqueName: \"kubernetes.io/projected/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-kube-api-access-nvjbq\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:43.121355 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.121361    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-maas-api-tls\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:43.121591 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.121375    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808-openshift-service-ca\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:43.476664 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.476624    2537 generic.go:356] "Generic (PLEG): container finished" podID="cb2dbdf8-a240-42f3-9a7c-391f9cb2a808" containerID="3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9" exitCode=0
Sep 17 14:36:43.477128 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.476674    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"
Sep 17 14:36:43.477128 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.476713    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m" event={"ID":"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808","Type":"ContainerDied","Data":"3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9"}
Sep 17 14:36:43.477128 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.476767    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m" event={"ID":"cb2dbdf8-a240-42f3-9a7c-391f9cb2a808","Type":"ContainerDied","Data":"f6db82afbb78c238207ccfd230555261650d56eaa2eeac81ef634733f0336d9c"}
Sep 17 14:36:43.477128 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.476794    2537 scope.go:117] "RemoveContainer" containerID="3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9"
Sep 17 14:36:43.478564 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.478532    2537 generic.go:356] "Generic (PLEG): container finished" podID="77d11546-5e73-4978-8161-5af243723c30" containerID="87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29" exitCode=0
Sep 17 14:36:43.478670 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.478616    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7" event={"ID":"77d11546-5e73-4978-8161-5af243723c30","Type":"ContainerDied","Data":"87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29"}
Sep 17 14:36:43.478670 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.478659    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7" event={"ID":"77d11546-5e73-4978-8161-5af243723c30","Type":"ContainerDied","Data":"79e99ee2ea2525543c8308d85c5701b40fb90e5280e1de2209b134e482d0b393"}
Sep 17 14:36:43.478791 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.478723    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"
Sep 17 14:36:43.487827 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.487809    2537 scope.go:117] "RemoveContainer" containerID="3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9"
Sep 17 14:36:43.488300 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:36:43.488270    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9\": container with ID starting with 3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9 not found: ID does not exist" containerID="3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9"
Sep 17 14:36:43.488383 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.488306    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9"} err="failed to get container status \"3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9\": rpc error: code = NotFound desc = could not find container \"3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9\": container with ID starting with 3d4c8484f67644afe2224010a17ba13de8d37462c629a9e08dea1d5116c695d9 not found: ID does not exist"
Sep 17 14:36:43.488383 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.488354    2537 scope.go:117] "RemoveContainer" containerID="87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29"
Sep 17 14:36:43.496829 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.496813    2537 scope.go:117] "RemoveContainer" containerID="87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29"
Sep 17 14:36:43.497111 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:36:43.497093    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29\": container with ID starting with 87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29 not found: ID does not exist" containerID="87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29"
Sep 17 14:36:43.497148 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.497119    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29"} err="failed to get container status \"87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29\": rpc error: code = NotFound desc = could not find container \"87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29\": container with ID starting with 87b724ce47618800e652a80f519912eebf242f6f0e993f84262463e4daa71e29 not found: ID does not exist"
Sep 17 14:36:43.501220 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.501171    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"]
Sep 17 14:36:43.504617 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.504592    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b50f2-9d546cd88-mdc6m"]
Sep 17 14:36:43.514831 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.514810    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"]
Sep 17 14:36:43.518783 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:43.518762    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b50f2-868dc945f8-qvlb7"]
Sep 17 14:36:44.245852 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:44.245820    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d11546-5e73-4978-8161-5af243723c30" path="/var/lib/kubelet/pods/77d11546-5e73-4978-8161-5af243723c30/volumes"
Sep 17 14:36:44.246235 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:44.246221    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb2dbdf8-a240-42f3-9a7c-391f9cb2a808" path="/var/lib/kubelet/pods/cb2dbdf8-a240-42f3-9a7c-391f9cb2a808/volumes"
Sep 17 14:36:46.800226 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:46.800141    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27" podUID="1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" containerName="main" containerID="cri-o://775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e" gracePeriod=45
Sep 17 14:36:46.812614 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:46.812584    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5" podUID="0b73f4fe-354f-4b09-b98d-509ba817198e" containerName="main" containerID="cri-o://7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641" gracePeriod=45
Sep 17 14:36:46.959822 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:46.959787    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27" podUID="1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 14:36:47.077312 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.077285    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:36:47.080078 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.080059    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:36:47.156414 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156385    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-home\") pod \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") "
Sep 17 14:36:47.156603 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156439    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z54l7\" (UniqueName: \"kubernetes.io/projected/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-kube-api-access-z54l7\") pod \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") "
Sep 17 14:36:47.156603 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156501    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96gcm\" (UniqueName: \"kubernetes.io/projected/0b73f4fe-354f-4b09-b98d-509ba817198e-kube-api-access-96gcm\") pod \"0b73f4fe-354f-4b09-b98d-509ba817198e\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") "
Sep 17 14:36:47.156603 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156526    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tls-certs\") pod \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") "
Sep 17 14:36:47.156603 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156549    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tmp-dir\") pod \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") "
Sep 17 14:36:47.156603 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156577    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-dshm\") pod \"0b73f4fe-354f-4b09-b98d-509ba817198e\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") "
Sep 17 14:36:47.156868 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156604    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-tmp-dir\") pod \"0b73f4fe-354f-4b09-b98d-509ba817198e\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") "
Sep 17 14:36:47.156868 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156631    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-home\") pod \"0b73f4fe-354f-4b09-b98d-509ba817198e\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") "
Sep 17 14:36:47.156868 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156682    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-home" (OuterVolumeSpecName: "home") pod "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" (UID: "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.156868 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156724    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-model-cache\") pod \"0b73f4fe-354f-4b09-b98d-509ba817198e\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") "
Sep 17 14:36:47.156868 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156757    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-dshm\") pod \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") "
Sep 17 14:36:47.156868 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156787    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-model-cache\") pod \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\" (UID: \"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8\") "
Sep 17 14:36:47.156868 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.156827    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b73f4fe-354f-4b09-b98d-509ba817198e-tls-certs\") pod \"0b73f4fe-354f-4b09-b98d-509ba817198e\" (UID: \"0b73f4fe-354f-4b09-b98d-509ba817198e\") "
Sep 17 14:36:47.157244 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.157051    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b73f4fe-354f-4b09-b98d-509ba817198e" (UID: "0b73f4fe-354f-4b09-b98d-509ba817198e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.157244 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.157203    2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-tmp-dir\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.157244 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.157222    2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-home\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.157690 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.157655    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-home" (OuterVolumeSpecName: "home") pod "0b73f4fe-354f-4b09-b98d-509ba817198e" (UID: "0b73f4fe-354f-4b09-b98d-509ba817198e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.157916 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.157892    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" (UID: "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.158004 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.157909    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" (UID: "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.158004 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.157943    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b73f4fe-354f-4b09-b98d-509ba817198e" (UID: "0b73f4fe-354f-4b09-b98d-509ba817198e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.159106 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.159064    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" (UID: "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:36:47.159282 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.159224    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-kube-api-access-z54l7" (OuterVolumeSpecName: "kube-api-access-z54l7") pod "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" (UID: "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8"). InnerVolumeSpecName "kube-api-access-z54l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:47.159549 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.159524    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-dshm" (OuterVolumeSpecName: "dshm") pod "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" (UID: "1bf58dc3-dcc6-40b5-8391-11cb03ee25e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.159825 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.159795    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b73f4fe-354f-4b09-b98d-509ba817198e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b73f4fe-354f-4b09-b98d-509ba817198e" (UID: "0b73f4fe-354f-4b09-b98d-509ba817198e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:36:47.159905 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.159845    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-dshm" (OuterVolumeSpecName: "dshm") pod "0b73f4fe-354f-4b09-b98d-509ba817198e" (UID: "0b73f4fe-354f-4b09-b98d-509ba817198e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.159905 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.159885    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b73f4fe-354f-4b09-b98d-509ba817198e-kube-api-access-96gcm" (OuterVolumeSpecName: "kube-api-access-96gcm") pod "0b73f4fe-354f-4b09-b98d-509ba817198e" (UID: "0b73f4fe-354f-4b09-b98d-509ba817198e"). InnerVolumeSpecName "kube-api-access-96gcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:47.258242 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.258207    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96gcm\" (UniqueName: \"kubernetes.io/projected/0b73f4fe-354f-4b09-b98d-509ba817198e-kube-api-access-96gcm\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.258242 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.258235    2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tls-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.258242 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.258246    2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-tmp-dir\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.258473 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.258254    2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-dshm\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.258473 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.258262    2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-home\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.258473 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.258270    2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b73f4fe-354f-4b09-b98d-509ba817198e-model-cache\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.258473 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.258277    2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-dshm\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.258473 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.258284    2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-model-cache\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.258473 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.258291    2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b73f4fe-354f-4b09-b98d-509ba817198e-tls-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.258473 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.258299    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z54l7\" (UniqueName: \"kubernetes.io/projected/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8-kube-api-access-z54l7\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.504672 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.503068    2537 generic.go:356] "Generic (PLEG): container finished" podID="1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" containerID="775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e" exitCode=0
Sep 17 14:36:47.504672 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.503313    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27" event={"ID":"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8","Type":"ContainerDied","Data":"775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e"}
Sep 17 14:36:47.504672 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.503357    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"
Sep 17 14:36:47.504672 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.503377    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27" event={"ID":"1bf58dc3-dcc6-40b5-8391-11cb03ee25e8","Type":"ContainerDied","Data":"94ab7c7bc4b826d53ed5699dc96335bdc6b9e457aa23392266fde095cafb3372"}
Sep 17 14:36:47.504672 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.503415    2537 scope.go:117] "RemoveContainer" containerID="775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e"
Sep 17 14:36:47.512085 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.512060    2537 generic.go:356] "Generic (PLEG): container finished" podID="0b73f4fe-354f-4b09-b98d-509ba817198e" containerID="7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641" exitCode=0
Sep 17 14:36:47.512281 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.512119    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5" event={"ID":"0b73f4fe-354f-4b09-b98d-509ba817198e","Type":"ContainerDied","Data":"7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641"}
Sep 17 14:36:47.512281 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.512150    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5" event={"ID":"0b73f4fe-354f-4b09-b98d-509ba817198e","Type":"ContainerDied","Data":"45f33d442ecb3c60e11213dabd30e8ae8bf0e84ef0ca7690cfd124c243099358"}
Sep 17 14:36:47.512281 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.512157    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"
Sep 17 14:36:47.520858 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.520838    2537 scope.go:117] "RemoveContainer" containerID="775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e"
Sep 17 14:36:47.521215 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:36:47.521165    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e\": container with ID starting with 775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e not found: ID does not exist" containerID="775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e"
Sep 17 14:36:47.521314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.521227    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e"} err="failed to get container status \"775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e\": rpc error: code = NotFound desc = could not find container \"775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e\": container with ID starting with 775f59eca85b2687428886847e12655cc85daeaad12ab934c002bc236538661e not found: ID does not exist"
Sep 17 14:36:47.521314 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.521250    2537 scope.go:117] "RemoveContainer" containerID="7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641"
Sep 17 14:36:47.529239 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.529221    2537 scope.go:117] "RemoveContainer" containerID="7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641"
Sep 17 14:36:47.529508 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:36:47.529487    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641\": container with ID starting with 7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641 not found: ID does not exist" containerID="7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641"
Sep 17 14:36:47.529582 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.529528    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641"} err="failed to get container status \"7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641\": rpc error: code = NotFound desc = could not find container \"7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641\": container with ID starting with 7123f1273cd8fb26b10062eb15bc90cdffa1d04facd99a853b37576d787a5641 not found: ID does not exist"
Sep 17 14:36:47.532356 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.532333    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"]
Sep 17 14:36:47.537431 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.537406    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-w4-8b50f2-kserve-69bb696c76-gbt27"]
Sep 17 14:36:47.550622 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.550598    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"]
Sep 17 14:36:47.554074 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:47.554052    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/premium-w4-8b50f2-kserve-56965fdb57-lvxh5"]
Sep 17 14:36:48.245325 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:48.245286    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b73f4fe-354f-4b09-b98d-509ba817198e" path="/var/lib/kubelet/pods/0b73f4fe-354f-4b09-b98d-509ba817198e/volumes"
Sep 17 14:36:48.245920 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:36:48.245856    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" path="/var/lib/kubelet/pods/1bf58dc3-dcc6-40b5-8391-11cb03ee25e8/volumes"
Sep 17 14:40:39.597691 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.597635    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"]
Sep 17 14:40:39.599039 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599013    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fdd105-fea0-4e42-81fa-b71474e5ecde" containerName="payload-pre-processing"
Sep 17 14:40:39.599203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599083    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fdd105-fea0-4e42-81fa-b71474e5ecde" containerName="payload-pre-processing"
Sep 17 14:40:39.599203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599105    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a5a5c8-0c81-4b59-b65a-5a0c072bfaea" containerName="maas-api"
Sep 17 14:40:39.599203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599114    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a5a5c8-0c81-4b59-b65a-5a0c072bfaea" containerName="maas-api"
Sep 17 14:40:39.599203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599126    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90fdaa68-ad7c-48ce-bb3e-7597a282ccd7" containerName="payload-processing"
Sep 17 14:40:39.599203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599135    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="90fdaa68-ad7c-48ce-bb3e-7597a282ccd7" containerName="payload-processing"
Sep 17 14:40:39.599203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599152    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" containerName="main"
Sep 17 14:40:39.599203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599159    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" containerName="main"
Sep 17 14:40:39.599203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599171    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca97e45d-8be0-417d-a657-436387f6b59b" containerName="payload-pre-processing"
Sep 17 14:40:39.599203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599201    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca97e45d-8be0-417d-a657-436387f6b59b" containerName="payload-pre-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599215    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567841f6-61d0-495d-9d76-efa5f9f00db7" containerName="maas-api"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599223    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="567841f6-61d0-495d-9d76-efa5f9f00db7" containerName="maas-api"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599231    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d11546-5e73-4978-8161-5af243723c30" containerName="payload-pre-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599239    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d11546-5e73-4978-8161-5af243723c30" containerName="payload-pre-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599251    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2dbdf8-a240-42f3-9a7c-391f9cb2a808" containerName="maas-api"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599260    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2dbdf8-a240-42f3-9a7c-391f9cb2a808" containerName="maas-api"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599269    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b73f4fe-354f-4b09-b98d-509ba817198e" containerName="main"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599279    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b73f4fe-354f-4b09-b98d-509ba817198e" containerName="main"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599296    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d972e9-2881-4707-bffd-98bab02b8f41" containerName="payload-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599305    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d972e9-2881-4707-bffd-98bab02b8f41" containerName="payload-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599321    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f49f99-f175-4557-b309-1a914175600b" containerName="payload-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599330    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f49f99-f175-4557-b309-1a914175600b" containerName="payload-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599421    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b73f4fe-354f-4b09-b98d-509ba817198e" containerName="main"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599433    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="567841f6-61d0-495d-9d76-efa5f9f00db7" containerName="maas-api"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599443    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d11546-5e73-4978-8161-5af243723c30" containerName="payload-pre-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599453    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf58dc3-dcc6-40b5-8391-11cb03ee25e8" containerName="main"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599463    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a5a5c8-0c81-4b59-b65a-5a0c072bfaea" containerName="maas-api"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599474    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="41fdd105-fea0-4e42-81fa-b71474e5ecde" containerName="payload-pre-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599482    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca97e45d-8be0-417d-a657-436387f6b59b" containerName="payload-pre-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599490    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d972e9-2881-4707-bffd-98bab02b8f41" containerName="payload-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599499    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="90fdaa68-ad7c-48ce-bb3e-7597a282ccd7" containerName="payload-processing"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599507    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb2dbdf8-a240-42f3-9a7c-391f9cb2a808" containerName="maas-api"
Sep 17 14:40:39.599653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.599515    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f49f99-f175-4557-b309-1a914175600b" containerName="payload-processing"
Sep 17 14:40:39.603167 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.603140    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:39.608520 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.606331    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5943ae-serving-cert\""
Sep 17 14:40:39.612869 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.612839    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"]
Sep 17 14:40:39.721459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.721423    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:39.721459 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.721461    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:39.721673 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.721508    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f1a320-02fe-4956-ae00-0d3f67ba70ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:39.721673 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.721541    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk6mf\" (UniqueName: \"kubernetes.io/projected/39f1a320-02fe-4956-ae00-0d3f67ba70ae-kube-api-access-wk6mf\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:39.822858 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.822817    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:39.822858 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.822856    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:39.823065 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.822880    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f1a320-02fe-4956-ae00-0d3f67ba70ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:39.823065 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.822900    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6mf\" (UniqueName: \"kubernetes.io/projected/39f1a320-02fe-4956-ae00-0d3f67ba70ae-kube-api-access-wk6mf\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:39.823065 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:40:39.822978    2537 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-serving-cert: secret "maas-api-e2e-ipp-b-5943ae-serving-cert" not found
Sep 17 14:40:39.823170 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:40:39.823087    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-maas-api-tls podName:39f1a320-02fe-4956-ae00-0d3f67ba70ae nodeName:}" failed. No retries permitted until 2026-09-17 14:40:40.323065715 +0000 UTC m=+2044.614179575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-maas-api-tls") pod "maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq" (UID: "39f1a320-02fe-4956-ae00-0d3f67ba70ae") : secret "maas-api-e2e-ipp-b-5943ae-serving-cert" not found
Sep 17 14:40:39.823643 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.823620    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f1a320-02fe-4956-ae00-0d3f67ba70ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:39.825395 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.825365    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:39.832035 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:39.832015    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6mf\" (UniqueName: \"kubernetes.io/projected/39f1a320-02fe-4956-ae00-0d3f67ba70ae-kube-api-access-wk6mf\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:40.328094 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:40.328060    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:40.330500 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:40.330475    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:40.520561 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:40.520521    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:40.796846 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:40.796819    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"]
Sep 17 14:40:40.799293 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:40:40.799262    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f1a320_02fe_4956_ae00_0d3f67ba70ae.slice/crio-1cd292170ee534b45f1abe75590c6d49f1f70c458bd6c3a749a49afd716fabee WatchSource:0}: Error finding container 1cd292170ee534b45f1abe75590c6d49f1f70c458bd6c3a749a49afd716fabee: Status 404 returned error can't find the container with id 1cd292170ee534b45f1abe75590c6d49f1f70c458bd6c3a749a49afd716fabee
Sep 17 14:40:40.800706 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:40.800688    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:40:41.616161 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:41.616122    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq" event={"ID":"39f1a320-02fe-4956-ae00-0d3f67ba70ae","Type":"ContainerStarted","Data":"ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7"}
Sep 17 14:40:41.616353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:41.616194    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq" event={"ID":"39f1a320-02fe-4956-ae00-0d3f67ba70ae","Type":"ContainerStarted","Data":"1cd292170ee534b45f1abe75590c6d49f1f70c458bd6c3a749a49afd716fabee"}
Sep 17 14:40:41.616353 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:41.616309    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:40:41.633755 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:41.633699    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq" podStartSLOduration=2.086691702 podStartE2EDuration="2.633680779s" totalImagesPullingTime="546.989077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:40:41.632643881 +0000 UTC m=+2045.923757742" watchObservedRunningTime="2026-09-17 14:40:41.633680779 +0000 UTC m=+2045.924794640"
Sep 17 14:40:47.627200 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:40:47.627148    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:41:07.109213 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:41:07.109091    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fkwcw_f5f4a040-2239-41eb-b7c7-c941c7a3e30c/payload-processing/0.log"
Sep 17 14:41:36.257770 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:41:36.257736    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:41:36.263109 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:41:36.263087    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:41:53.615566 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:41:53.615532    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fkwcw_f5f4a040-2239-41eb-b7c7-c941c7a3e30c/payload-processing/0.log"
Sep 17 14:42:02.384742 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.384703    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"]
Sep 17 14:42:02.388488 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.388467    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:02.391246 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.391220    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-8fb478-serving-cert\""
Sep 17 14:42:02.401582 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.401556    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"]
Sep 17 14:42:02.442090 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.442060    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:02.442275 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.442096    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpv5m\" (UniqueName: \"kubernetes.io/projected/422a4397-1304-486a-9626-140cdb4c9b7f-kube-api-access-xpv5m\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:02.442275 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.442132    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/422a4397-1304-486a-9626-140cdb4c9b7f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:02.442275 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.442159    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:02.542632 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.542596    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:02.542632 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.542635    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpv5m\" (UniqueName: \"kubernetes.io/projected/422a4397-1304-486a-9626-140cdb4c9b7f-kube-api-access-xpv5m\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:02.542861 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.542668    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/422a4397-1304-486a-9626-140cdb4c9b7f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:02.542861 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.542771    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:02.542973 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:42:02.542913    2537 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-serving-cert: secret "maas-api-e2e-ipp-cleanup-8fb478-serving-cert" not found
Sep 17 14:42:02.543026 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:42:02.542998    2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-maas-api-tls podName:422a4397-1304-486a-9626-140cdb4c9b7f nodeName:}" failed. No retries permitted until 2026-09-17 14:42:03.042975628 +0000 UTC m=+2127.334089475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx" (UID: "422a4397-1304-486a-9626-140cdb4c9b7f") : secret "maas-api-e2e-ipp-cleanup-8fb478-serving-cert" not found
Sep 17 14:42:02.543345 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.543325    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/422a4397-1304-486a-9626-140cdb4c9b7f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:02.551019 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.550990    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:02.555492 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:02.555468    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpv5m\" (UniqueName: \"kubernetes.io/projected/422a4397-1304-486a-9626-140cdb4c9b7f-kube-api-access-xpv5m\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:03.047145 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.047108    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:03.049518 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.049500    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:03.071562 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.071533    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"]
Sep 17 14:42:03.075157 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.075137    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:03.080679 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.080661    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8fb478\""
Sep 17 14:42:03.095059 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.095032    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"]
Sep 17 14:42:03.147556 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.147523    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnxqq\" (UniqueName: \"kubernetes.io/projected/dd9faa21-7164-461b-aaaf-5e29785cac66-kube-api-access-tnxqq\") pod \"payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k\" (UID: \"dd9faa21-7164-461b-aaaf-5e29785cac66\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:03.147556 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.147555    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9faa21-7164-461b-aaaf-5e29785cac66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k\" (UID: \"dd9faa21-7164-461b-aaaf-5e29785cac66\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:03.248102 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.248065    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnxqq\" (UniqueName: \"kubernetes.io/projected/dd9faa21-7164-461b-aaaf-5e29785cac66-kube-api-access-tnxqq\") pod \"payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k\" (UID: \"dd9faa21-7164-461b-aaaf-5e29785cac66\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:03.248102 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.248102    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9faa21-7164-461b-aaaf-5e29785cac66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k\" (UID: \"dd9faa21-7164-461b-aaaf-5e29785cac66\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:03.248877 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.248856    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9faa21-7164-461b-aaaf-5e29785cac66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k\" (UID: \"dd9faa21-7164-461b-aaaf-5e29785cac66\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:03.258516 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.258489    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnxqq\" (UniqueName: \"kubernetes.io/projected/dd9faa21-7164-461b-aaaf-5e29785cac66-kube-api-access-tnxqq\") pod \"payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k\" (UID: \"dd9faa21-7164-461b-aaaf-5e29785cac66\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:03.299344 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.299268    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:03.385595 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.385549    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:03.611398 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.611375    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"]
Sep 17 14:42:03.613656 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:42:03.613626    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422a4397_1304_486a_9626_140cdb4c9b7f.slice/crio-317fc2a1332bac98b888dc0617031669ecc0d9b253035e5fa3b5cb812f57d16f WatchSource:0}: Error finding container 317fc2a1332bac98b888dc0617031669ecc0d9b253035e5fa3b5cb812f57d16f: Status 404 returned error can't find the container with id 317fc2a1332bac98b888dc0617031669ecc0d9b253035e5fa3b5cb812f57d16f
Sep 17 14:42:03.671558 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:03.671529    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"]
Sep 17 14:42:03.676889 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:42:03.676861    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9faa21_7164_461b_aaaf_5e29785cac66.slice/crio-289e59e74479305b7c13ab3c15d2708a0402f8767044b219091181cd1b4bef9d WatchSource:0}: Error finding container 289e59e74479305b7c13ab3c15d2708a0402f8767044b219091181cd1b4bef9d: Status 404 returned error can't find the container with id 289e59e74479305b7c13ab3c15d2708a0402f8767044b219091181cd1b4bef9d
Sep 17 14:42:04.005451 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:04.005358    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx" event={"ID":"422a4397-1304-486a-9626-140cdb4c9b7f","Type":"ContainerStarted","Data":"317fc2a1332bac98b888dc0617031669ecc0d9b253035e5fa3b5cb812f57d16f"}
Sep 17 14:42:04.006871 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:04.006842    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k" event={"ID":"dd9faa21-7164-461b-aaaf-5e29785cac66","Type":"ContainerStarted","Data":"74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577"}
Sep 17 14:42:04.006996 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:04.006879    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k" event={"ID":"dd9faa21-7164-461b-aaaf-5e29785cac66","Type":"ContainerStarted","Data":"289e59e74479305b7c13ab3c15d2708a0402f8767044b219091181cd1b4bef9d"}
Sep 17 14:42:04.006996 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:04.006927    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:04.045751 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:04.045705    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k" podStartSLOduration=1.045688113 podStartE2EDuration="1.045688113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:42:04.044296657 +0000 UTC m=+2128.335410520" watchObservedRunningTime="2026-09-17 14:42:04.045688113 +0000 UTC m=+2128.336801974"
Sep 17 14:42:05.013108 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:05.013068    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx" event={"ID":"422a4397-1304-486a-9626-140cdb4c9b7f","Type":"ContainerStarted","Data":"1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27"}
Sep 17 14:42:06.018693 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:06.018654    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:12.029189 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:12.029147    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:12.052157 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:12.052103    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx" podStartSLOduration=9.532718863 podStartE2EDuration="10.052086842s" totalImagesPullingTime="519.367979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:42:05.070691826 +0000 UTC m=+2129.361805687" watchObservedRunningTime="2026-09-17 14:42:12.052086842 +0000 UTC m=+2136.343200702"
Sep 17 14:42:15.015875 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:15.015836    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:40.292893 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.292810    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"]
Sep 17 14:42:40.293341 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.293035    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx" podUID="422a4397-1304-486a-9626-140cdb4c9b7f" containerName="maas-api" containerID="cri-o://1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27" gracePeriod=30
Sep 17 14:42:40.356138 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.356090    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"]
Sep 17 14:42:40.356438 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.356400    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k" podUID="dd9faa21-7164-461b-aaaf-5e29785cac66" containerName="payload-processing" containerID="cri-o://74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577" gracePeriod=30
Sep 17 14:42:40.612152 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.612061    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:40.650320 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.650292    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:40.787349 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.787311    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpv5m\" (UniqueName: \"kubernetes.io/projected/422a4397-1304-486a-9626-140cdb4c9b7f-kube-api-access-xpv5m\") pod \"422a4397-1304-486a-9626-140cdb4c9b7f\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") "
Sep 17 14:42:40.787529 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.787364    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-maas-api-tls\") pod \"422a4397-1304-486a-9626-140cdb4c9b7f\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") "
Sep 17 14:42:40.787529 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.787384    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9faa21-7164-461b-aaaf-5e29785cac66-plugins-config-volume\") pod \"dd9faa21-7164-461b-aaaf-5e29785cac66\" (UID: \"dd9faa21-7164-461b-aaaf-5e29785cac66\") "
Sep 17 14:42:40.787529 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.787428    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-metrics-certs\") pod \"422a4397-1304-486a-9626-140cdb4c9b7f\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") "
Sep 17 14:42:40.787529 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.787450    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/422a4397-1304-486a-9626-140cdb4c9b7f-openshift-service-ca\") pod \"422a4397-1304-486a-9626-140cdb4c9b7f\" (UID: \"422a4397-1304-486a-9626-140cdb4c9b7f\") "
Sep 17 14:42:40.787529 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.787506    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnxqq\" (UniqueName: \"kubernetes.io/projected/dd9faa21-7164-461b-aaaf-5e29785cac66-kube-api-access-tnxqq\") pod \"dd9faa21-7164-461b-aaaf-5e29785cac66\" (UID: \"dd9faa21-7164-461b-aaaf-5e29785cac66\") "
Sep 17 14:42:40.787853 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.787826    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9faa21-7164-461b-aaaf-5e29785cac66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd9faa21-7164-461b-aaaf-5e29785cac66" (UID: "dd9faa21-7164-461b-aaaf-5e29785cac66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:42:40.787915 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.787864    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422a4397-1304-486a-9626-140cdb4c9b7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "422a4397-1304-486a-9626-140cdb4c9b7f" (UID: "422a4397-1304-486a-9626-140cdb4c9b7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:42:40.789600 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.789577    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422a4397-1304-486a-9626-140cdb4c9b7f-kube-api-access-xpv5m" (OuterVolumeSpecName: "kube-api-access-xpv5m") pod "422a4397-1304-486a-9626-140cdb4c9b7f" (UID: "422a4397-1304-486a-9626-140cdb4c9b7f"). InnerVolumeSpecName "kube-api-access-xpv5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:42:40.789685 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.789640    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "422a4397-1304-486a-9626-140cdb4c9b7f" (UID: "422a4397-1304-486a-9626-140cdb4c9b7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:42:40.789850 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.789824    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "422a4397-1304-486a-9626-140cdb4c9b7f" (UID: "422a4397-1304-486a-9626-140cdb4c9b7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:42:40.789955 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.789877    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9faa21-7164-461b-aaaf-5e29785cac66-kube-api-access-tnxqq" (OuterVolumeSpecName: "kube-api-access-tnxqq") pod "dd9faa21-7164-461b-aaaf-5e29785cac66" (UID: "dd9faa21-7164-461b-aaaf-5e29785cac66"). InnerVolumeSpecName "kube-api-access-tnxqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:42:40.888203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.888148    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-maas-api-tls\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:42:40.888203 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.888202    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9faa21-7164-461b-aaaf-5e29785cac66-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:42:40.888412 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.888219    2537 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/422a4397-1304-486a-9626-140cdb4c9b7f-metrics-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:42:40.888412 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.888233    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/422a4397-1304-486a-9626-140cdb4c9b7f-openshift-service-ca\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:42:40.888412 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.888244    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnxqq\" (UniqueName: \"kubernetes.io/projected/dd9faa21-7164-461b-aaaf-5e29785cac66-kube-api-access-tnxqq\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:42:40.888412 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:40.888255    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpv5m\" (UniqueName: \"kubernetes.io/projected/422a4397-1304-486a-9626-140cdb4c9b7f-kube-api-access-xpv5m\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:42:41.185205 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.185102    2537 generic.go:356] "Generic (PLEG): container finished" podID="dd9faa21-7164-461b-aaaf-5e29785cac66" containerID="74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577" exitCode=0
Sep 17 14:42:41.185205 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.185158    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"
Sep 17 14:42:41.185446 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.185194    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k" event={"ID":"dd9faa21-7164-461b-aaaf-5e29785cac66","Type":"ContainerDied","Data":"74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577"}
Sep 17 14:42:41.185446 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.185285    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k" event={"ID":"dd9faa21-7164-461b-aaaf-5e29785cac66","Type":"ContainerDied","Data":"289e59e74479305b7c13ab3c15d2708a0402f8767044b219091181cd1b4bef9d"}
Sep 17 14:42:41.185446 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.185316    2537 scope.go:117] "RemoveContainer" containerID="74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577"
Sep 17 14:42:41.186631 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.186608    2537 generic.go:356] "Generic (PLEG): container finished" podID="422a4397-1304-486a-9626-140cdb4c9b7f" containerID="1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27" exitCode=0
Sep 17 14:42:41.186734 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.186663    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx" event={"ID":"422a4397-1304-486a-9626-140cdb4c9b7f","Type":"ContainerDied","Data":"1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27"}
Sep 17 14:42:41.186734 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.186667    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"
Sep 17 14:42:41.186734 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.186706    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx" event={"ID":"422a4397-1304-486a-9626-140cdb4c9b7f","Type":"ContainerDied","Data":"317fc2a1332bac98b888dc0617031669ecc0d9b253035e5fa3b5cb812f57d16f"}
Sep 17 14:42:41.193613 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.193583    2537 scope.go:117] "RemoveContainer" containerID="74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577"
Sep 17 14:42:41.193867 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:42:41.193846    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577\": container with ID starting with 74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577 not found: ID does not exist" containerID="74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577"
Sep 17 14:42:41.193923 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.193874    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577"} err="failed to get container status \"74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577\": rpc error: code = NotFound desc = could not find container \"74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577\": container with ID starting with 74a707441cb9960fd46142309cec542459d307c9a2333c5b54ff4ffd5d9d6577 not found: ID does not exist"
Sep 17 14:42:41.193923 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.193889    2537 scope.go:117] "RemoveContainer" containerID="1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27"
Sep 17 14:42:41.202339 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.202320    2537 scope.go:117] "RemoveContainer" containerID="1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27"
Sep 17 14:42:41.202590 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:42:41.202572    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27\": container with ID starting with 1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27 not found: ID does not exist" containerID="1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27"
Sep 17 14:42:41.202653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.202595    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27"} err="failed to get container status \"1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27\": rpc error: code = NotFound desc = could not find container \"1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27\": container with ID starting with 1f923f9c91716cc57ac62f182b1f9b1ddb1249627ddb22572f87f67c2f8b0d27 not found: ID does not exist"
Sep 17 14:42:41.222837 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.222813    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"]
Sep 17 14:42:41.247416 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.247389    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8fb478-6865d49749-7fq7k"]
Sep 17 14:42:41.268572 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.268544    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"]
Sep 17 14:42:41.271654 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:41.271631    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8fb478-6c48f4f9fb-k4bgx"]
Sep 17 14:42:42.245301 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:42.245270    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="422a4397-1304-486a-9626-140cdb4c9b7f" path="/var/lib/kubelet/pods/422a4397-1304-486a-9626-140cdb4c9b7f/volumes"
Sep 17 14:42:42.245698 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:42:42.245635    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd9faa21-7164-461b-aaaf-5e29785cac66" path="/var/lib/kubelet/pods/dd9faa21-7164-461b-aaaf-5e29785cac66/volumes"
Sep 17 14:44:07.270711 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.270674    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"]
Sep 17 14:44:07.271310 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.270900    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq" podUID="39f1a320-02fe-4956-ae00-0d3f67ba70ae" containerName="maas-api" containerID="cri-o://ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7" gracePeriod=30
Sep 17 14:44:07.520606 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.520582    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:44:07.601502 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.601469    2537 generic.go:356] "Generic (PLEG): container finished" podID="39f1a320-02fe-4956-ae00-0d3f67ba70ae" containerID="ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7" exitCode=0
Sep 17 14:44:07.601647 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.601513    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"
Sep 17 14:44:07.601647 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.601535    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq" event={"ID":"39f1a320-02fe-4956-ae00-0d3f67ba70ae","Type":"ContainerDied","Data":"ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7"}
Sep 17 14:44:07.601647 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.601598    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq" event={"ID":"39f1a320-02fe-4956-ae00-0d3f67ba70ae","Type":"ContainerDied","Data":"1cd292170ee534b45f1abe75590c6d49f1f70c458bd6c3a749a49afd716fabee"}
Sep 17 14:44:07.601647 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.601627    2537 scope.go:117] "RemoveContainer" containerID="ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7"
Sep 17 14:44:07.610946 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.610921    2537 scope.go:117] "RemoveContainer" containerID="ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7"
Sep 17 14:44:07.611233 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:44:07.611216    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7\": container with ID starting with ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7 not found: ID does not exist" containerID="ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7"
Sep 17 14:44:07.611292 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.611241    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7"} err="failed to get container status \"ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7\": rpc error: code = NotFound desc = could not find container \"ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7\": container with ID starting with ff7f9bbb3bc3dbaf2739a008001324d575917d08a014eef5c2262443df6617c7 not found: ID does not exist"
Sep 17 14:44:07.634700 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.634677    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-maas-api-tls\") pod \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") "
Sep 17 14:44:07.634825 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.634707    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f1a320-02fe-4956-ae00-0d3f67ba70ae-openshift-service-ca\") pod \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") "
Sep 17 14:44:07.634825 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.634758    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk6mf\" (UniqueName: \"kubernetes.io/projected/39f1a320-02fe-4956-ae00-0d3f67ba70ae-kube-api-access-wk6mf\") pod \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") "
Sep 17 14:44:07.634825 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.634795    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-metrics-certs\") pod \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\" (UID: \"39f1a320-02fe-4956-ae00-0d3f67ba70ae\") "
Sep 17 14:44:07.635107 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.635082    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f1a320-02fe-4956-ae00-0d3f67ba70ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39f1a320-02fe-4956-ae00-0d3f67ba70ae" (UID: "39f1a320-02fe-4956-ae00-0d3f67ba70ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:07.636876 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.636846    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "39f1a320-02fe-4956-ae00-0d3f67ba70ae" (UID: "39f1a320-02fe-4956-ae00-0d3f67ba70ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:44:07.637142 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.637113    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f1a320-02fe-4956-ae00-0d3f67ba70ae-kube-api-access-wk6mf" (OuterVolumeSpecName: "kube-api-access-wk6mf") pod "39f1a320-02fe-4956-ae00-0d3f67ba70ae" (UID: "39f1a320-02fe-4956-ae00-0d3f67ba70ae"). InnerVolumeSpecName "kube-api-access-wk6mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:07.637245 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.637111    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39f1a320-02fe-4956-ae00-0d3f67ba70ae" (UID: "39f1a320-02fe-4956-ae00-0d3f67ba70ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:44:07.735702 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.735668    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-maas-api-tls\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:07.735702 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.735698    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f1a320-02fe-4956-ae00-0d3f67ba70ae-openshift-service-ca\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:07.735702 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.735707    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk6mf\" (UniqueName: \"kubernetes.io/projected/39f1a320-02fe-4956-ae00-0d3f67ba70ae-kube-api-access-wk6mf\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:07.735936 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.735716    2537 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f1a320-02fe-4956-ae00-0d3f67ba70ae-metrics-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:07.924053 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.924021    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"]
Sep 17 14:44:07.927510 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:07.927487    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5943ae-6757f5bcdd-2shvq"]
Sep 17 14:44:08.245866 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:08.245791    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39f1a320-02fe-4956-ae00-0d3f67ba70ae" path="/var/lib/kubelet/pods/39f1a320-02fe-4956-ae00-0d3f67ba70ae/volumes"
Sep 17 14:44:25.492236 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.492203    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"]
Sep 17 14:44:25.495551 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.492421    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj" podUID="797e0e0c-1e96-47a9-b5dd-505facd731b4" containerName="maas-api" containerID="cri-o://70fb7b349d21785ac8a8c467ae99012f622d3bfddbd108551a2e8f9016fbfe6d" gracePeriod=30
Sep 17 14:44:25.566667 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.566629    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"]
Sep 17 14:44:25.566944 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.566915    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj" podUID="2426d0ee-79d8-4edc-94cd-1128816cdbd4" containerName="payload-processing" containerID="cri-o://37620d838f4808152669402a8067a63b8420d3314eeeee716d1e680511c958ac" gracePeriod=30
Sep 17 14:44:25.688079 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.688050    2537 generic.go:356] "Generic (PLEG): container finished" podID="2426d0ee-79d8-4edc-94cd-1128816cdbd4" containerID="37620d838f4808152669402a8067a63b8420d3314eeeee716d1e680511c958ac" exitCode=0
Sep 17 14:44:25.688288 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.688121    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj" event={"ID":"2426d0ee-79d8-4edc-94cd-1128816cdbd4","Type":"ContainerDied","Data":"37620d838f4808152669402a8067a63b8420d3314eeeee716d1e680511c958ac"}
Sep 17 14:44:25.690418 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.690387    2537 generic.go:356] "Generic (PLEG): container finished" podID="797e0e0c-1e96-47a9-b5dd-505facd731b4" containerID="70fb7b349d21785ac8a8c467ae99012f622d3bfddbd108551a2e8f9016fbfe6d" exitCode=0
Sep 17 14:44:25.690575 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.690474    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj" event={"ID":"797e0e0c-1e96-47a9-b5dd-505facd731b4","Type":"ContainerDied","Data":"70fb7b349d21785ac8a8c467ae99012f622d3bfddbd108551a2e8f9016fbfe6d"}
Sep 17 14:44:25.818918 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.818894    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:44:25.849437 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.849404    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:44:25.894746 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.894715    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-maas-api-tls\") pod \"797e0e0c-1e96-47a9-b5dd-505facd731b4\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") "
Sep 17 14:44:25.894915 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.894765    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45s5l\" (UniqueName: \"kubernetes.io/projected/797e0e0c-1e96-47a9-b5dd-505facd731b4-kube-api-access-45s5l\") pod \"797e0e0c-1e96-47a9-b5dd-505facd731b4\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") "
Sep 17 14:44:25.894915 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.894793    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-metrics-certs\") pod \"797e0e0c-1e96-47a9-b5dd-505facd731b4\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") "
Sep 17 14:44:25.894915 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.894825    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797e0e0c-1e96-47a9-b5dd-505facd731b4-openshift-service-ca\") pod \"797e0e0c-1e96-47a9-b5dd-505facd731b4\" (UID: \"797e0e0c-1e96-47a9-b5dd-505facd731b4\") "
Sep 17 14:44:25.895291 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.895262    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797e0e0c-1e96-47a9-b5dd-505facd731b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "797e0e0c-1e96-47a9-b5dd-505facd731b4" (UID: "797e0e0c-1e96-47a9-b5dd-505facd731b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:25.896805 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.896776    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "797e0e0c-1e96-47a9-b5dd-505facd731b4" (UID: "797e0e0c-1e96-47a9-b5dd-505facd731b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:44:25.897323 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.897305    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "797e0e0c-1e96-47a9-b5dd-505facd731b4" (UID: "797e0e0c-1e96-47a9-b5dd-505facd731b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:44:25.897394 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.897345    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797e0e0c-1e96-47a9-b5dd-505facd731b4-kube-api-access-45s5l" (OuterVolumeSpecName: "kube-api-access-45s5l") pod "797e0e0c-1e96-47a9-b5dd-505facd731b4" (UID: "797e0e0c-1e96-47a9-b5dd-505facd731b4"). InnerVolumeSpecName "kube-api-access-45s5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:25.995490 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.995457    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2426d0ee-79d8-4edc-94cd-1128816cdbd4-plugins-config-volume\") pod \"2426d0ee-79d8-4edc-94cd-1128816cdbd4\" (UID: \"2426d0ee-79d8-4edc-94cd-1128816cdbd4\") "
Sep 17 14:44:25.995647 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.995586    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmntz\" (UniqueName: \"kubernetes.io/projected/2426d0ee-79d8-4edc-94cd-1128816cdbd4-kube-api-access-jmntz\") pod \"2426d0ee-79d8-4edc-94cd-1128816cdbd4\" (UID: \"2426d0ee-79d8-4edc-94cd-1128816cdbd4\") "
Sep 17 14:44:25.995715 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.995704    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-maas-api-tls\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:25.995758 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.995718    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45s5l\" (UniqueName: \"kubernetes.io/projected/797e0e0c-1e96-47a9-b5dd-505facd731b4-kube-api-access-45s5l\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:25.995758 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.995727    2537 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797e0e0c-1e96-47a9-b5dd-505facd731b4-metrics-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:25.995758 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.995736    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797e0e0c-1e96-47a9-b5dd-505facd731b4-openshift-service-ca\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:25.995854 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.995808    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2426d0ee-79d8-4edc-94cd-1128816cdbd4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2426d0ee-79d8-4edc-94cd-1128816cdbd4" (UID: "2426d0ee-79d8-4edc-94cd-1128816cdbd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:25.997536 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:25.997516    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2426d0ee-79d8-4edc-94cd-1128816cdbd4-kube-api-access-jmntz" (OuterVolumeSpecName: "kube-api-access-jmntz") pod "2426d0ee-79d8-4edc-94cd-1128816cdbd4" (UID: "2426d0ee-79d8-4edc-94cd-1128816cdbd4"). InnerVolumeSpecName "kube-api-access-jmntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:26.096485 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.096412    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmntz\" (UniqueName: \"kubernetes.io/projected/2426d0ee-79d8-4edc-94cd-1128816cdbd4-kube-api-access-jmntz\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:26.096485 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.096439    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2426d0ee-79d8-4edc-94cd-1128816cdbd4-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:26.696376 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.696346    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"
Sep 17 14:44:26.696376 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.696355    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj" event={"ID":"2426d0ee-79d8-4edc-94cd-1128816cdbd4","Type":"ContainerDied","Data":"3ee40739bc6557ba122ea601469a8efd866c49e3e1be3b4ddfe58c6d14d781c2"}
Sep 17 14:44:26.696852 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.696432    2537 scope.go:117] "RemoveContainer" containerID="37620d838f4808152669402a8067a63b8420d3314eeeee716d1e680511c958ac"
Sep 17 14:44:26.698087 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.698067    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"
Sep 17 14:44:26.698251 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.698066    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj" event={"ID":"797e0e0c-1e96-47a9-b5dd-505facd731b4","Type":"ContainerDied","Data":"40b52c72dcec2ee661a71f65abbb616eb51e3653cfbc66caaa82307b6cdfdebf"}
Sep 17 14:44:26.705333 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.705248    2537 scope.go:117] "RemoveContainer" containerID="70fb7b349d21785ac8a8c467ae99012f622d3bfddbd108551a2e8f9016fbfe6d"
Sep 17 14:44:26.722250 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.722227    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"]
Sep 17 14:44:26.728416 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.728395    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3da4be-6db8d59d69-fgqmj"]
Sep 17 14:44:26.745798 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.745774    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"]
Sep 17 14:44:26.758044 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:26.757985    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3da4be-5c8b766b8b-tq8fj"]
Sep 17 14:44:28.250666 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:28.250627    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2426d0ee-79d8-4edc-94cd-1128816cdbd4" path="/var/lib/kubelet/pods/2426d0ee-79d8-4edc-94cd-1128816cdbd4/volumes"
Sep 17 14:44:28.251193 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:28.251159    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="797e0e0c-1e96-47a9-b5dd-505facd731b4" path="/var/lib/kubelet/pods/797e0e0c-1e96-47a9-b5dd-505facd731b4/volumes"
Sep 17 14:44:43.617323 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:43.617282    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"]
Sep 17 14:44:43.622654 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:43.622598    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"]
Sep 17 14:44:43.622904 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:43.622876    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4" podUID="fd0b1b85-0f3a-447f-a2ea-5dd4077974d6" containerName="payload-pre-processing" containerID="cri-o://451f88d81485eff4112bfd4322327542b40e17c255d8c0a0960be4178a89756b" gracePeriod=30
Sep 17 14:44:43.779143 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:43.779106    2537 generic.go:356] "Generic (PLEG): container finished" podID="fd0b1b85-0f3a-447f-a2ea-5dd4077974d6" containerID="451f88d81485eff4112bfd4322327542b40e17c255d8c0a0960be4178a89756b" exitCode=0
Sep 17 14:44:43.779337 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:43.779203    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4" event={"ID":"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6","Type":"ContainerDied","Data":"451f88d81485eff4112bfd4322327542b40e17c255d8c0a0960be4178a89756b"}
Sep 17 14:44:43.779424 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:43.779402    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp" podUID="13a1932a-e1dd-4444-9b05-7dd008bd2d33" containerName="payload-processing" containerID="cri-o://acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f" gracePeriod=30
Sep 17 14:44:43.902481 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:43.902454    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:44:43.944549 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:43.944513    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lchgx\" (UniqueName: \"kubernetes.io/projected/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-kube-api-access-lchgx\") pod \"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6\" (UID: \"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6\") "
Sep 17 14:44:43.944741 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:43.944615    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-plugins-config-volume\") pod \"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6\" (UID: \"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6\") "
Sep 17 14:44:43.944990 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:43.944966    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd0b1b85-0f3a-447f-a2ea-5dd4077974d6" (UID: "fd0b1b85-0f3a-447f-a2ea-5dd4077974d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:43.947055 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:43.947027    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-kube-api-access-lchgx" (OuterVolumeSpecName: "kube-api-access-lchgx") pod "fd0b1b85-0f3a-447f-a2ea-5dd4077974d6" (UID: "fd0b1b85-0f3a-447f-a2ea-5dd4077974d6"). InnerVolumeSpecName "kube-api-access-lchgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:44.026820 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.026794    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:44:44.045142 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.045116    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvcl7\" (UniqueName: \"kubernetes.io/projected/13a1932a-e1dd-4444-9b05-7dd008bd2d33-kube-api-access-bvcl7\") pod \"13a1932a-e1dd-4444-9b05-7dd008bd2d33\" (UID: \"13a1932a-e1dd-4444-9b05-7dd008bd2d33\") "
Sep 17 14:44:44.045330 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.045170    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a1932a-e1dd-4444-9b05-7dd008bd2d33-plugins-config-volume\") pod \"13a1932a-e1dd-4444-9b05-7dd008bd2d33\" (UID: \"13a1932a-e1dd-4444-9b05-7dd008bd2d33\") "
Sep 17 14:44:44.045415 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.045394    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:44.045471 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.045416    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lchgx\" (UniqueName: \"kubernetes.io/projected/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6-kube-api-access-lchgx\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:44.045571 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.045542    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a1932a-e1dd-4444-9b05-7dd008bd2d33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13a1932a-e1dd-4444-9b05-7dd008bd2d33" (UID: "13a1932a-e1dd-4444-9b05-7dd008bd2d33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:44.047132 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.047115    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a1932a-e1dd-4444-9b05-7dd008bd2d33-kube-api-access-bvcl7" (OuterVolumeSpecName: "kube-api-access-bvcl7") pod "13a1932a-e1dd-4444-9b05-7dd008bd2d33" (UID: "13a1932a-e1dd-4444-9b05-7dd008bd2d33"). InnerVolumeSpecName "kube-api-access-bvcl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:44.146085 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.145999    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvcl7\" (UniqueName: \"kubernetes.io/projected/13a1932a-e1dd-4444-9b05-7dd008bd2d33-kube-api-access-bvcl7\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:44.146085 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.146030    2537 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a1932a-e1dd-4444-9b05-7dd008bd2d33-plugins-config-volume\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:44.784866 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.784833    2537 generic.go:356] "Generic (PLEG): container finished" podID="13a1932a-e1dd-4444-9b05-7dd008bd2d33" containerID="acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f" exitCode=0
Sep 17 14:44:44.785368 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.784885    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"
Sep 17 14:44:44.785368 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.784909    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp" event={"ID":"13a1932a-e1dd-4444-9b05-7dd008bd2d33","Type":"ContainerDied","Data":"acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f"}
Sep 17 14:44:44.785368 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.784962    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp" event={"ID":"13a1932a-e1dd-4444-9b05-7dd008bd2d33","Type":"ContainerDied","Data":"5122ef5ecb83cf12f57951f78235753f54e3396905aa663ccaf356af4abe929a"}
Sep 17 14:44:44.785368 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.784990    2537 scope.go:117] "RemoveContainer" containerID="acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f"
Sep 17 14:44:44.786360 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.786340    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4" event={"ID":"fd0b1b85-0f3a-447f-a2ea-5dd4077974d6","Type":"ContainerDied","Data":"3660d736ed16ef9aa78e6d10d4872f562675304d18a19a3c6e274f447f473602"}
Sep 17 14:44:44.786428 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.786365    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"
Sep 17 14:44:44.793555 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.793534    2537 scope.go:117] "RemoveContainer" containerID="acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f"
Sep 17 14:44:44.793880 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:44:44.793846    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f\": container with ID starting with acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f not found: ID does not exist" containerID="acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f"
Sep 17 14:44:44.793980 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.793879    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f"} err="failed to get container status \"acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f\": rpc error: code = NotFound desc = could not find container \"acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f\": container with ID starting with acb4e29c96c4c964e955f71144c08201649e89e970c6a09752c2d6134c25711f not found: ID does not exist"
Sep 17 14:44:44.793980 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.793901    2537 scope.go:117] "RemoveContainer" containerID="451f88d81485eff4112bfd4322327542b40e17c255d8c0a0960be4178a89756b"
Sep 17 14:44:44.806315 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.806294    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"]
Sep 17 14:44:44.812422 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.812402    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b48b7c-599955fc9d-r7tf4"]
Sep 17 14:44:44.825299 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.825278    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"]
Sep 17 14:44:44.830095 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:44.830072    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b48b7c-77f6777557-27dvp"]
Sep 17 14:44:46.244797 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:46.244765    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a1932a-e1dd-4444-9b05-7dd008bd2d33" path="/var/lib/kubelet/pods/13a1932a-e1dd-4444-9b05-7dd008bd2d33/volumes"
Sep 17 14:44:46.245164 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:46.245116    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd0b1b85-0f3a-447f-a2ea-5dd4077974d6" path="/var/lib/kubelet/pods/fd0b1b85-0f3a-447f-a2ea-5dd4077974d6/volumes"
Sep 17 14:44:57.554252 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.554214    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-4b4sj"]
Sep 17 14:44:57.554710 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.554476    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj" podUID="7b259c42-97a1-49f2-a4b7-48631ebc0a08" containerName="manager" containerID="cri-o://e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94" gracePeriod=30
Sep 17 14:44:57.794638 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.794614    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:44:57.849232 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.849116    2537 generic.go:356] "Generic (PLEG): container finished" podID="7b259c42-97a1-49f2-a4b7-48631ebc0a08" containerID="e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94" exitCode=0
Sep 17 14:44:57.849232 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.849169    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj"
Sep 17 14:44:57.849387 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.849191    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj" event={"ID":"7b259c42-97a1-49f2-a4b7-48631ebc0a08","Type":"ContainerDied","Data":"e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94"}
Sep 17 14:44:57.849387 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.849308    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6d9dbbc-4b4sj" event={"ID":"7b259c42-97a1-49f2-a4b7-48631ebc0a08","Type":"ContainerDied","Data":"3852820e51d97efd6932aa590033db1646dfaa3d61ca7eb60ba540c79e0fa13c"}
Sep 17 14:44:57.849387 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.849332    2537 scope.go:117] "RemoveContainer" containerID="e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94"
Sep 17 14:44:57.849975 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.849956    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-webhook-cert\") pod \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") "
Sep 17 14:44:57.850075 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.849991    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p28pg\" (UniqueName: \"kubernetes.io/projected/7b259c42-97a1-49f2-a4b7-48631ebc0a08-kube-api-access-p28pg\") pod \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") "
Sep 17 14:44:57.850075 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.850038    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-metrics-certs\") pod \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\" (UID: \"7b259c42-97a1-49f2-a4b7-48631ebc0a08\") "
Sep 17 14:44:57.852469 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.852424    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7b259c42-97a1-49f2-a4b7-48631ebc0a08" (UID: "7b259c42-97a1-49f2-a4b7-48631ebc0a08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:44:57.852690 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.852654    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7b259c42-97a1-49f2-a4b7-48631ebc0a08" (UID: "7b259c42-97a1-49f2-a4b7-48631ebc0a08"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:44:57.852810 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.852782    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b259c42-97a1-49f2-a4b7-48631ebc0a08-kube-api-access-p28pg" (OuterVolumeSpecName: "kube-api-access-p28pg") pod "7b259c42-97a1-49f2-a4b7-48631ebc0a08" (UID: "7b259c42-97a1-49f2-a4b7-48631ebc0a08"). InnerVolumeSpecName "kube-api-access-p28pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:57.869056 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.869035    2537 scope.go:117] "RemoveContainer" containerID="e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94"
Sep 17 14:44:57.869349 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:44:57.869332    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94\": container with ID starting with e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94 not found: ID does not exist" containerID="e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94"
Sep 17 14:44:57.869422 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.869357    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94"} err="failed to get container status \"e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94\": rpc error: code = NotFound desc = could not find container \"e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94\": container with ID starting with e2dbdddf64b859047b03d7b517117c271394557cd9315b7d5ca614d4ee5b7f94 not found: ID does not exist"
Sep 17 14:44:57.951165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.951134    2537 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-webhook-cert\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:57.951165 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.951166    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p28pg\" (UniqueName: \"kubernetes.io/projected/7b259c42-97a1-49f2-a4b7-48631ebc0a08-kube-api-access-p28pg\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:57.951400 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:57.951203    2537 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b259c42-97a1-49f2-a4b7-48631ebc0a08-metrics-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:44:58.175347 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:58.175315    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-4b4sj"]
Sep 17 14:44:58.178864 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:58.178836    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-4b4sj"]
Sep 17 14:44:58.246035 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:44:58.245992    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b259c42-97a1-49f2-a4b7-48631ebc0a08" path="/var/lib/kubelet/pods/7b259c42-97a1-49f2-a4b7-48631ebc0a08/volumes"
Sep 17 14:45:58.992094 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992058    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-q2t9m"]
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992395    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2426d0ee-79d8-4edc-94cd-1128816cdbd4" containerName="payload-processing"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992408    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="2426d0ee-79d8-4edc-94cd-1128816cdbd4" containerName="payload-processing"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992417    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="797e0e0c-1e96-47a9-b5dd-505facd731b4" containerName="maas-api"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992422    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="797e0e0c-1e96-47a9-b5dd-505facd731b4" containerName="maas-api"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992436    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd0b1b85-0f3a-447f-a2ea-5dd4077974d6" containerName="payload-pre-processing"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992442    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd0b1b85-0f3a-447f-a2ea-5dd4077974d6" containerName="payload-pre-processing"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992450    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a1932a-e1dd-4444-9b05-7dd008bd2d33" containerName="payload-processing"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992455    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a1932a-e1dd-4444-9b05-7dd008bd2d33" containerName="payload-processing"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992463    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9faa21-7164-461b-aaaf-5e29785cac66" containerName="payload-processing"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992468    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9faa21-7164-461b-aaaf-5e29785cac66" containerName="payload-processing"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992477    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422a4397-1304-486a-9626-140cdb4c9b7f" containerName="maas-api"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992483    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="422a4397-1304-486a-9626-140cdb4c9b7f" containerName="maas-api"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992490    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f1a320-02fe-4956-ae00-0d3f67ba70ae" containerName="maas-api"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992495    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f1a320-02fe-4956-ae00-0d3f67ba70ae" containerName="maas-api"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992502    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b259c42-97a1-49f2-a4b7-48631ebc0a08" containerName="manager"
Sep 17 14:45:58.992541 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992507    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b259c42-97a1-49f2-a4b7-48631ebc0a08" containerName="manager"
Sep 17 14:45:58.993023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992555    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="797e0e0c-1e96-47a9-b5dd-505facd731b4" containerName="maas-api"
Sep 17 14:45:58.993023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992562    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd0b1b85-0f3a-447f-a2ea-5dd4077974d6" containerName="payload-pre-processing"
Sep 17 14:45:58.993023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992568    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b259c42-97a1-49f2-a4b7-48631ebc0a08" containerName="manager"
Sep 17 14:45:58.993023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992573    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a1932a-e1dd-4444-9b05-7dd008bd2d33" containerName="payload-processing"
Sep 17 14:45:58.993023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992579    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="2426d0ee-79d8-4edc-94cd-1128816cdbd4" containerName="payload-processing"
Sep 17 14:45:58.993023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992586    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f1a320-02fe-4956-ae00-0d3f67ba70ae" containerName="maas-api"
Sep 17 14:45:58.993023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992591    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd9faa21-7164-461b-aaaf-5e29785cac66" containerName="payload-processing"
Sep 17 14:45:58.993023 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.992597    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="422a4397-1304-486a-9626-140cdb4c9b7f" containerName="maas-api"
Sep 17 14:45:58.995393 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:58.995374    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:45:59.000057 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.000020    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 14:45:59.000057 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.000048    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 14:45:59.000293 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.000120    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hnczm\""
Sep 17 14:45:59.000293 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.000125    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 14:45:59.010247 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.010218    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-q2t9m"]
Sep 17 14:45:59.058087 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.058056    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-metrics-certs\") pod \"maas-controller-6bf6d9dbbc-q2t9m\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") " pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:45:59.058274 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.058103    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4sd\" (UniqueName: \"kubernetes.io/projected/41099755-031b-4ad6-a3af-ae0241723406-kube-api-access-mp4sd\") pod \"maas-controller-6bf6d9dbbc-q2t9m\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") " pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:45:59.058274 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.058122    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-webhook-cert\") pod \"maas-controller-6bf6d9dbbc-q2t9m\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") " pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:45:59.159084 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.159049    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4sd\" (UniqueName: \"kubernetes.io/projected/41099755-031b-4ad6-a3af-ae0241723406-kube-api-access-mp4sd\") pod \"maas-controller-6bf6d9dbbc-q2t9m\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") " pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:45:59.159084 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.159085    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-webhook-cert\") pod \"maas-controller-6bf6d9dbbc-q2t9m\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") " pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:45:59.159309 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.159141    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-metrics-certs\") pod \"maas-controller-6bf6d9dbbc-q2t9m\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") " pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:45:59.161681 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.161652    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-metrics-certs\") pod \"maas-controller-6bf6d9dbbc-q2t9m\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") " pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:45:59.161681 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.161668    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-webhook-cert\") pod \"maas-controller-6bf6d9dbbc-q2t9m\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") " pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:45:59.170079 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.170057    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4sd\" (UniqueName: \"kubernetes.io/projected/41099755-031b-4ad6-a3af-ae0241723406-kube-api-access-mp4sd\") pod \"maas-controller-6bf6d9dbbc-q2t9m\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") " pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:45:59.305136 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.305048    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:45:59.583421 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.583399    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-q2t9m"]
Sep 17 14:45:59.585698 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:45:59.585666    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41099755_031b_4ad6_a3af_ae0241723406.slice/crio-33ce6a064e3d4b4732a7701d451880d6c480f0a1e220b5908bb35fba462e9118 WatchSource:0}: Error finding container 33ce6a064e3d4b4732a7701d451880d6c480f0a1e220b5908bb35fba462e9118: Status 404 returned error can't find the container with id 33ce6a064e3d4b4732a7701d451880d6c480f0a1e220b5908bb35fba462e9118
Sep 17 14:45:59.587105 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:45:59.587082    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:46:00.129619 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:00.129583    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m" event={"ID":"41099755-031b-4ad6-a3af-ae0241723406","Type":"ContainerStarted","Data":"f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd"}
Sep 17 14:46:00.130087 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:00.129637    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m" event={"ID":"41099755-031b-4ad6-a3af-ae0241723406","Type":"ContainerStarted","Data":"33ce6a064e3d4b4732a7701d451880d6c480f0a1e220b5908bb35fba462e9118"}
Sep 17 14:46:00.130087 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:00.129681    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:46:00.159920 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:00.159872    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m" podStartSLOduration=1.708593339 podStartE2EDuration="2.159856382s" totalImagesPullingTime="451.263043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:46:00.158146327 +0000 UTC m=+2364.449260189" watchObservedRunningTime="2026-09-17 14:46:00.159856382 +0000 UTC m=+2364.450970269"
Sep 17 14:46:11.136057 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:11.136027    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:46:20.463275 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:20.463232    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-q2t9m"]
Sep 17 14:46:20.463762 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:20.463511    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m" podUID="41099755-031b-4ad6-a3af-ae0241723406" containerName="manager" containerID="cri-o://f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd" gracePeriod=30
Sep 17 14:46:21.203549 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.203521    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:46:21.226986 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.226946    2537 generic.go:356] "Generic (PLEG): container finished" podID="41099755-031b-4ad6-a3af-ae0241723406" containerID="f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd" exitCode=0
Sep 17 14:46:21.227164 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.227000    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m"
Sep 17 14:46:21.227164 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.226993    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m" event={"ID":"41099755-031b-4ad6-a3af-ae0241723406","Type":"ContainerDied","Data":"f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd"}
Sep 17 14:46:21.227164 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.227130    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m" event={"ID":"41099755-031b-4ad6-a3af-ae0241723406","Type":"ContainerDied","Data":"33ce6a064e3d4b4732a7701d451880d6c480f0a1e220b5908bb35fba462e9118"}
Sep 17 14:46:21.227164 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.227161    2537 scope.go:117] "RemoveContainer" containerID="f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd"
Sep 17 14:46:21.236953 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.236936    2537 scope.go:117] "RemoveContainer" containerID="f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd"
Sep 17 14:46:21.237297 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:46:21.237276    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd\": container with ID starting with f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd not found: ID does not exist" containerID="f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd"
Sep 17 14:46:21.237365 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.237307    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd"} err="failed to get container status \"f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd\": rpc error: code = NotFound desc = could not find container \"f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd\": container with ID starting with f6a8cdcc97d73e33f51c7896c7f19ca1557de7b9ba2fe95ac51daee2f7eff8bd not found: ID does not exist"
Sep 17 14:46:21.351014 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.350984    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-webhook-cert\") pod \"41099755-031b-4ad6-a3af-ae0241723406\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") "
Sep 17 14:46:21.351198 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.351082    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-metrics-certs\") pod \"41099755-031b-4ad6-a3af-ae0241723406\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") "
Sep 17 14:46:21.351198 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.351100    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp4sd\" (UniqueName: \"kubernetes.io/projected/41099755-031b-4ad6-a3af-ae0241723406-kube-api-access-mp4sd\") pod \"41099755-031b-4ad6-a3af-ae0241723406\" (UID: \"41099755-031b-4ad6-a3af-ae0241723406\") "
Sep 17 14:46:21.353085 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.353053    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "41099755-031b-4ad6-a3af-ae0241723406" (UID: "41099755-031b-4ad6-a3af-ae0241723406"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:46:21.353224 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.353098    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "41099755-031b-4ad6-a3af-ae0241723406" (UID: "41099755-031b-4ad6-a3af-ae0241723406"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:46:21.353224 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.353131    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41099755-031b-4ad6-a3af-ae0241723406-kube-api-access-mp4sd" (OuterVolumeSpecName: "kube-api-access-mp4sd") pod "41099755-031b-4ad6-a3af-ae0241723406" (UID: "41099755-031b-4ad6-a3af-ae0241723406"). InnerVolumeSpecName "kube-api-access-mp4sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:46:21.452260 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.452224    2537 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-webhook-cert\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:46:21.452260 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.452253    2537 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41099755-031b-4ad6-a3af-ae0241723406-metrics-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:46:21.452260 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.452263    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4sd\" (UniqueName: \"kubernetes.io/projected/41099755-031b-4ad6-a3af-ae0241723406-kube-api-access-mp4sd\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:46:21.549926 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.549891    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-q2t9m"]
Sep 17 14:46:21.555142 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:21.555116    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-q2t9m"]
Sep 17 14:46:22.135052 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.135003    2537 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-6bf6d9dbbc-q2t9m" podUID="41099755-031b-4ad6-a3af-ae0241723406" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.63:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 14:46:22.244418 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.244384    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41099755-031b-4ad6-a3af-ae0241723406" path="/var/lib/kubelet/pods/41099755-031b-4ad6-a3af-ae0241723406/volumes"
Sep 17 14:46:22.353620 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.353589    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-5jnd8"]
Sep 17 14:46:22.353944 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.353931    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41099755-031b-4ad6-a3af-ae0241723406" containerName="manager"
Sep 17 14:46:22.354008 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.353946    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="41099755-031b-4ad6-a3af-ae0241723406" containerName="manager"
Sep 17 14:46:22.354008 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.353998    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="41099755-031b-4ad6-a3af-ae0241723406" containerName="manager"
Sep 17 14:46:22.358284 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.358268    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:22.366514 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.366465    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hnczm\""
Sep 17 14:46:22.366665 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.366511    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 14:46:22.366665 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.366560    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 14:46:22.366665 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.366597    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 14:46:22.387786 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.387724    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-5jnd8"]
Sep 17 14:46:22.459371 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.459334    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rsvm\" (UniqueName: \"kubernetes.io/projected/91c72e06-3799-48c4-b06d-eba49a8e997d-kube-api-access-2rsvm\") pod \"maas-controller-6bf6d9dbbc-5jnd8\" (UID: \"91c72e06-3799-48c4-b06d-eba49a8e997d\") " pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:22.459515 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.459379    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91c72e06-3799-48c4-b06d-eba49a8e997d-metrics-certs\") pod \"maas-controller-6bf6d9dbbc-5jnd8\" (UID: \"91c72e06-3799-48c4-b06d-eba49a8e997d\") " pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:22.459515 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.459427    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91c72e06-3799-48c4-b06d-eba49a8e997d-webhook-cert\") pod \"maas-controller-6bf6d9dbbc-5jnd8\" (UID: \"91c72e06-3799-48c4-b06d-eba49a8e997d\") " pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:22.560851 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.560814    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rsvm\" (UniqueName: \"kubernetes.io/projected/91c72e06-3799-48c4-b06d-eba49a8e997d-kube-api-access-2rsvm\") pod \"maas-controller-6bf6d9dbbc-5jnd8\" (UID: \"91c72e06-3799-48c4-b06d-eba49a8e997d\") " pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:22.561266 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.560860    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91c72e06-3799-48c4-b06d-eba49a8e997d-metrics-certs\") pod \"maas-controller-6bf6d9dbbc-5jnd8\" (UID: \"91c72e06-3799-48c4-b06d-eba49a8e997d\") " pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:22.561266 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.560879    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91c72e06-3799-48c4-b06d-eba49a8e997d-webhook-cert\") pod \"maas-controller-6bf6d9dbbc-5jnd8\" (UID: \"91c72e06-3799-48c4-b06d-eba49a8e997d\") " pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:22.563290 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.563267    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91c72e06-3799-48c4-b06d-eba49a8e997d-webhook-cert\") pod \"maas-controller-6bf6d9dbbc-5jnd8\" (UID: \"91c72e06-3799-48c4-b06d-eba49a8e997d\") " pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:22.563410 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.563299    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91c72e06-3799-48c4-b06d-eba49a8e997d-metrics-certs\") pod \"maas-controller-6bf6d9dbbc-5jnd8\" (UID: \"91c72e06-3799-48c4-b06d-eba49a8e997d\") " pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:22.571996 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.571971    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rsvm\" (UniqueName: \"kubernetes.io/projected/91c72e06-3799-48c4-b06d-eba49a8e997d-kube-api-access-2rsvm\") pod \"maas-controller-6bf6d9dbbc-5jnd8\" (UID: \"91c72e06-3799-48c4-b06d-eba49a8e997d\") " pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:22.668573 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.668477    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:22.937936 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:22.937911    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bf6d9dbbc-5jnd8"]
Sep 17 14:46:22.939881 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:46:22.939851    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c72e06_3799_48c4_b06d_eba49a8e997d.slice/crio-de5b89e32c9f79feda70eb1313f2475c54318b2ffd94d7f388a4c25e592e7406 WatchSource:0}: Error finding container de5b89e32c9f79feda70eb1313f2475c54318b2ffd94d7f388a4c25e592e7406: Status 404 returned error can't find the container with id de5b89e32c9f79feda70eb1313f2475c54318b2ffd94d7f388a4c25e592e7406
Sep 17 14:46:23.238856 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:23.238774    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8" event={"ID":"91c72e06-3799-48c4-b06d-eba49a8e997d","Type":"ContainerStarted","Data":"de5b89e32c9f79feda70eb1313f2475c54318b2ffd94d7f388a4c25e592e7406"}
Sep 17 14:46:24.247667 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:24.247635    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8" event={"ID":"91c72e06-3799-48c4-b06d-eba49a8e997d","Type":"ContainerStarted","Data":"009c795daf1377c36c6b0755d0bb12d7f5fda6442d9aaaff14c321f2ff872862"}
Sep 17 14:46:24.248022 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:24.247693    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:24.274679 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:24.274626    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8" podStartSLOduration=1.943531173 podStartE2EDuration="2.274612128s" totalImagesPullingTime="331.080955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:46:24.274169702 +0000 UTC m=+2388.565283565" watchObservedRunningTime="2026-09-17 14:46:24.274612128 +0000 UTC m=+2388.565725989"
Sep 17 14:46:35.253482 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:35.253451    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bf6d9dbbc-5jnd8"
Sep 17 14:46:36.283495 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:36.283462    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:46:36.290265 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:46:36.290239    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:51:06.838369 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.838335    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"]
Sep 17 14:51:06.841658 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.841638    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:06.854087 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.854061    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"]
Sep 17 14:51:06.892894 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.892861    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39ba0167-9fdc-408e-9ca5-7e6250cb5708-maas-api-tls\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:06.893041 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.892910    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tglkg\" (UniqueName: \"kubernetes.io/projected/39ba0167-9fdc-408e-9ca5-7e6250cb5708-kube-api-access-tglkg\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:06.893041 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.892997    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39ba0167-9fdc-408e-9ca5-7e6250cb5708-openshift-service-ca\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:06.893138 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.893055    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39ba0167-9fdc-408e-9ca5-7e6250cb5708-metrics-certs\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:06.994207 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.994158    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39ba0167-9fdc-408e-9ca5-7e6250cb5708-metrics-certs\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:06.994207 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.994212    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39ba0167-9fdc-408e-9ca5-7e6250cb5708-maas-api-tls\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:06.994409 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.994240    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tglkg\" (UniqueName: \"kubernetes.io/projected/39ba0167-9fdc-408e-9ca5-7e6250cb5708-kube-api-access-tglkg\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:06.994409 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.994280    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39ba0167-9fdc-408e-9ca5-7e6250cb5708-openshift-service-ca\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:06.995156 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.995135    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39ba0167-9fdc-408e-9ca5-7e6250cb5708-openshift-service-ca\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:06.996878 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.996852    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39ba0167-9fdc-408e-9ca5-7e6250cb5708-maas-api-tls\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:06.996977 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:06.996897    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39ba0167-9fdc-408e-9ca5-7e6250cb5708-metrics-certs\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:07.002608 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:07.002587    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tglkg\" (UniqueName: \"kubernetes.io/projected/39ba0167-9fdc-408e-9ca5-7e6250cb5708-kube-api-access-tglkg\") pod \"maas-api-6f6d65c644-dcnlj\" (UID: \"39ba0167-9fdc-408e-9ca5-7e6250cb5708\") " pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:07.151901 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:07.151855    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:07.432785 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:07.432757    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"]
Sep 17 14:51:07.435516 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:51:07.435488    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ba0167_9fdc_408e_9ca5_7e6250cb5708.slice/crio-bf8a5a3cbed26bc5771169431f6c037f5b8841d0bf88eb611ba9491a72793402 WatchSource:0}: Error finding container bf8a5a3cbed26bc5771169431f6c037f5b8841d0bf88eb611ba9491a72793402: Status 404 returned error can't find the container with id bf8a5a3cbed26bc5771169431f6c037f5b8841d0bf88eb611ba9491a72793402
Sep 17 14:51:07.437262 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:07.437237    2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:51:07.492385 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:07.492348    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj" event={"ID":"39ba0167-9fdc-408e-9ca5-7e6250cb5708","Type":"ContainerStarted","Data":"bf8a5a3cbed26bc5771169431f6c037f5b8841d0bf88eb611ba9491a72793402"}
Sep 17 14:51:08.498521 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:08.498480    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj" event={"ID":"39ba0167-9fdc-408e-9ca5-7e6250cb5708","Type":"ContainerStarted","Data":"15a5a1234ac0b96052d856b014432b1444c5bb5f46aef2853e3518e56cceb9ac"}
Sep 17 14:51:08.498890 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:08.498546    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:14.508698 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:14.508666    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj"
Sep 17 14:51:14.527572 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:14.527522    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f6d65c644-dcnlj" podStartSLOduration=8.106177444 podStartE2EDuration="8.527502844s" totalImagesPullingTime="421.3254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:51:08.544973665 +0000 UTC m=+2672.836087526" watchObservedRunningTime="2026-09-17 14:51:14.527502844 +0000 UTC m=+2678.818616706"
Sep 17 14:51:14.557817 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:14.557765    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"]
Sep 17 14:51:14.558063 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:14.558015    2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s" podUID="66670616-aaa0-4758-a736-446f4e654409" containerName="maas-api" containerID="cri-o://9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f" gracePeriod=30
Sep 17 14:51:15.800480 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.800452    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:51:15.869032 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.868998    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66670616-aaa0-4758-a736-446f4e654409-openshift-service-ca\") pod \"66670616-aaa0-4758-a736-446f4e654409\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") "
Sep 17 14:51:15.869222 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.869079    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-maas-api-tls\") pod \"66670616-aaa0-4758-a736-446f4e654409\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") "
Sep 17 14:51:15.869222 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.869109    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-metrics-certs\") pod \"66670616-aaa0-4758-a736-446f4e654409\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") "
Sep 17 14:51:15.869222 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.869148    2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fd9w\" (UniqueName: \"kubernetes.io/projected/66670616-aaa0-4758-a736-446f4e654409-kube-api-access-6fd9w\") pod \"66670616-aaa0-4758-a736-446f4e654409\" (UID: \"66670616-aaa0-4758-a736-446f4e654409\") "
Sep 17 14:51:15.869373 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.869348    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66670616-aaa0-4758-a736-446f4e654409-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66670616-aaa0-4758-a736-446f4e654409" (UID: "66670616-aaa0-4758-a736-446f4e654409"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:51:15.871375 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.871346    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66670616-aaa0-4758-a736-446f4e654409" (UID: "66670616-aaa0-4758-a736-446f4e654409"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:51:15.871375 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.871358    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "66670616-aaa0-4758-a736-446f4e654409" (UID: "66670616-aaa0-4758-a736-446f4e654409"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:51:15.871564 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.871364    2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66670616-aaa0-4758-a736-446f4e654409-kube-api-access-6fd9w" (OuterVolumeSpecName: "kube-api-access-6fd9w") pod "66670616-aaa0-4758-a736-446f4e654409" (UID: "66670616-aaa0-4758-a736-446f4e654409"). InnerVolumeSpecName "kube-api-access-6fd9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:51:15.970493 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.970410    2537 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-maas-api-tls\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:51:15.970493 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.970441    2537 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66670616-aaa0-4758-a736-446f4e654409-metrics-certs\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:51:15.970493 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.970452    2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fd9w\" (UniqueName: \"kubernetes.io/projected/66670616-aaa0-4758-a736-446f4e654409-kube-api-access-6fd9w\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:51:15.970493 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:15.970464    2537 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66670616-aaa0-4758-a736-446f4e654409-openshift-service-ca\") on node \"ip-10-0-133-98.ec2.internal\" DevicePath \"\""
Sep 17 14:51:16.538839 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:16.538749    2537 generic.go:356] "Generic (PLEG): container finished" podID="66670616-aaa0-4758-a736-446f4e654409" containerID="9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f" exitCode=0
Sep 17 14:51:16.538839 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:16.538786    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s" event={"ID":"66670616-aaa0-4758-a736-446f4e654409","Type":"ContainerDied","Data":"9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f"}
Sep 17 14:51:16.538839 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:16.538823    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s" event={"ID":"66670616-aaa0-4758-a736-446f4e654409","Type":"ContainerDied","Data":"79457e04a92cd4b0afb83e7e63bfc80b9460649ea7055c59fff90d0f5d15a2e1"}
Sep 17 14:51:16.538839 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:16.538843    2537 scope.go:117] "RemoveContainer" containerID="9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f"
Sep 17 14:51:16.539122 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:16.538788    2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"
Sep 17 14:51:16.548821 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:16.548796    2537 scope.go:117] "RemoveContainer" containerID="9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f"
Sep 17 14:51:16.549082 ip-10-0-133-98 kubenswrapper[2537]: E0917 14:51:16.549062    2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f\": container with ID starting with 9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f not found: ID does not exist" containerID="9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f"
Sep 17 14:51:16.549142 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:16.549090    2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f"} err="failed to get container status \"9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f\": rpc error: code = NotFound desc = could not find container \"9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f\": container with ID starting with 9a00ddc5b38ce3ecf21b4ff7635b6e44b282b74dd4d93587d177964f5894e23f not found: ID does not exist"
Sep 17 14:51:16.557147 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:16.557121    2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"]
Sep 17 14:51:16.560874 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:16.560850    2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9b6644d-r8s7s"]
Sep 17 14:51:18.248382 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:18.245792    2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66670616-aaa0-4758-a736-446f4e654409" path="/var/lib/kubelet/pods/66670616-aaa0-4758-a736-446f4e654409/volumes"
Sep 17 14:51:36.308596 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:36.308528    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:51:36.316162 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:36.316131    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fn7m_e47ece57-c1f5-425d-81f9-6f255a586ec0/ovn-acl-logging/0.log"
Sep 17 14:51:41.667098 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:41.667057    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-j4c8c_3164cd21-60ad-4914-ac6f-769b3489289e/manager/0.log"
Sep 17 14:51:41.904566 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:41.904536    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bf6d9dbbc-5jnd8_91c72e06-3799-48c4-b06d-eba49a8e997d/manager/0.log"
Sep 17 14:51:42.016601 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:42.016513    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ccvnw_98afef09-5ea8-49e4-b0fb-b11305bddd31/server/0.log"
Sep 17 14:51:42.132766 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:42.132738    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hxhh6_c47cc71d-8718-4874-a0ed-8f1fb3e53be0/manager/0.log"
Sep 17 14:51:42.374670 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:42.374646    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-569db487f9-q92qv_fe7032d6-ba74-471f-90e2-b539862c19ec/copy-manifests/0.log"
Sep 17 14:51:42.381475 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:42.381440    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-569db487f9-q92qv_fe7032d6-ba74-471f-90e2-b539862c19ec/manager/0.log"
Sep 17 14:51:42.928410 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:42.928378    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f6d65c644-dcnlj_39ba0167-9fdc-408e-9ca5-7e6250cb5708/maas-api/0.log"
Sep 17 14:51:43.254907 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:43.254825    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-9zdbz_ceab7c52-fb1e-499e-bf5f-7081684a0558/postgres/0.log"
Sep 17 14:51:44.120233 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:44.120201    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-4hmzt_aa5f0f44-e2ab-4245-aa4e-2dd45d732988/manager/0.log"
Sep 17 14:51:44.224161 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:44.224118    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-wmq5r_6fe52c70-f73c-4698-bd86-2fa571da0689/manager/0.log"
Sep 17 14:51:45.137237 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:45.137197    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2fmw5_76bc7a14-41f9-4205-89c6-3a3a4f61cbe5/discovery/0.log"
Sep 17 14:51:45.243531 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:45.243493    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-549869464b-7pvhb_73a95568-9bbe-47dc-a8ea-de9b6a5eff04/kube-auth-proxy/0.log"
Sep 17 14:51:45.587007 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:45.586924    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-q95h6_c9df4506-8383-4582-92b0-b8c06c843697/payload-pre-processing/0.log"
Sep 17 14:51:45.707427 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:45.707390    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fkwcw_f5f4a040-2239-41eb-b7c7-c941c7a3e30c/payload-processing/0.log"
Sep 17 14:51:46.362334 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:46.362296    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-jx5jf_d8328e56-c364-4b6a-b9f7-2b70765bd1a4/main/0.log"
Sep 17 14:51:46.587778 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:46.587750    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f58xs5_607044ce-9261-46f4-a8ae-d8749056957f/main/0.log"
Sep 17 14:51:46.807879 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:46.807790    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-bpb2k_ff1c19f2-a335-4432-a231-f61094c134bb/main/0.log"
Sep 17 14:51:55.411426 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:55.411390    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8dc5c_16a9d26e-dfaa-4330-be84-a026c41d6f03/global-pull-secret-syncer/0.log"
Sep 17 14:51:55.535966 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:55.535933    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h78nv_d1d14525-b7fc-4b75-bae7-a22fbca657f3/konnectivity-agent/0.log"
Sep 17 14:51:55.625894 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:51:55.625867    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-98.ec2.internal_b6e3318f8bcf0aa565c80ccdb4bd6a32/haproxy/0.log"
Sep 17 14:52:00.531547 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:00.531516    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-4hmzt_aa5f0f44-e2ab-4245-aa4e-2dd45d732988/manager/0.log"
Sep 17 14:52:00.555687 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:00.555651    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-wmq5r_6fe52c70-f73c-4698-bd86-2fa571da0689/manager/0.log"
Sep 17 14:52:02.600117 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:02.600087    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bwhn7_edd8cf01-23e4-4cca-bce4-52210d98b048/node-exporter/0.log"
Sep 17 14:52:02.620101 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:02.620067    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bwhn7_edd8cf01-23e4-4cca-bce4-52210d98b048/kube-rbac-proxy/0.log"
Sep 17 14:52:02.641193 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:02.641148    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bwhn7_edd8cf01-23e4-4cca-bce4-52210d98b048/init-textfile/0.log"
Sep 17 14:52:05.493978 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.493945    2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"]
Sep 17 14:52:05.494370 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.494294    2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66670616-aaa0-4758-a736-446f4e654409" containerName="maas-api"
Sep 17 14:52:05.494370 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.494307    2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="66670616-aaa0-4758-a736-446f4e654409" containerName="maas-api"
Sep 17 14:52:05.494444 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.494389    2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="66670616-aaa0-4758-a736-446f4e654409" containerName="maas-api"
Sep 17 14:52:05.497448 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.497424    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.500530 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.500509    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xrpqc\"/\"kube-root-ca.crt\""
Sep 17 14:52:05.500653 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.500511    2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xrpqc\"/\"openshift-service-ca.crt\""
Sep 17 14:52:05.501797 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.501779    2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xrpqc\"/\"default-dockercfg-vgnzx\""
Sep 17 14:52:05.508329 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.508308    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"]
Sep 17 14:52:05.593677 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.593636    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-proc\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.593677 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.593679    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-lib-modules\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.593900 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.593762    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-244s4\" (UniqueName: \"kubernetes.io/projected/1e1bf49d-511b-4504-b769-5bdbe3b0488e-kube-api-access-244s4\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.593900 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.593802    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-podres\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.593900 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.593861    2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-sys\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.694773 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.694736    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-244s4\" (UniqueName: \"kubernetes.io/projected/1e1bf49d-511b-4504-b769-5bdbe3b0488e-kube-api-access-244s4\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.694947 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.694780    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-podres\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.694947 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.694840    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-sys\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.694947 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.694879    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-proc\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.694947 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.694912    2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-lib-modules\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.695154 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.694952    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-sys\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.695154 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.695002    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-podres\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.695154 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.695010    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-proc\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.695154 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.695063    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e1bf49d-511b-4504-b769-5bdbe3b0488e-lib-modules\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.703549 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.703522    2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-244s4\" (UniqueName: \"kubernetes.io/projected/1e1bf49d-511b-4504-b769-5bdbe3b0488e-kube-api-access-244s4\") pod \"perf-node-gather-daemonset-6fbhq\" (UID: \"1e1bf49d-511b-4504-b769-5bdbe3b0488e\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:05.808464 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:05.808387    2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:06.080142 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:06.080104    2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"]
Sep 17 14:52:06.082509 ip-10-0-133-98 kubenswrapper[2537]: W0917 14:52:06.082473    2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1e1bf49d_511b_4504_b769_5bdbe3b0488e.slice/crio-ebc48e54483a5eeeffb995cd8080e9cf651c287d32b4aab80e5d2fd2d4f04c6e WatchSource:0}: Error finding container ebc48e54483a5eeeffb995cd8080e9cf651c287d32b4aab80e5d2fd2d4f04c6e: Status 404 returned error can't find the container with id ebc48e54483a5eeeffb995cd8080e9cf651c287d32b4aab80e5d2fd2d4f04c6e
Sep 17 14:52:06.760844 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:06.760812    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq" event={"ID":"1e1bf49d-511b-4504-b769-5bdbe3b0488e","Type":"ContainerStarted","Data":"4801701e8fb069266c600a23517242f05745c00567aab0f4268078d12859a6a0"}
Sep 17 14:52:06.761230 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:06.760862    2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq" event={"ID":"1e1bf49d-511b-4504-b769-5bdbe3b0488e","Type":"ContainerStarted","Data":"ebc48e54483a5eeeffb995cd8080e9cf651c287d32b4aab80e5d2fd2d4f04c6e"}
Sep 17 14:52:06.761230 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:06.760893    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:06.787976 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:06.787930    2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq" podStartSLOduration=1.787913946 podStartE2EDuration="1.787913946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:52:06.786200677 +0000 UTC m=+2731.077314538" watchObservedRunningTime="2026-09-17 14:52:06.787913946 +0000 UTC m=+2731.079028110"
Sep 17 14:52:08.094890 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:08.094862    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m4qxw_05376a4a-6b6c-46cf-b8aa-77f1e8995a27/dns/0.log"
Sep 17 14:52:08.120306 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:08.120282    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m4qxw_05376a4a-6b6c-46cf-b8aa-77f1e8995a27/kube-rbac-proxy/0.log"
Sep 17 14:52:08.147264 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:08.147240    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8fgsv_d2bdc7d6-aa6b-4234-aa41-4b4fbf8d5d03/dns-node-resolver/0.log"
Sep 17 14:52:08.837682 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:08.837649    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xz96l_f856d63c-e51d-4532-a4e1-73fc32c4c203/node-ca/0.log"
Sep 17 14:52:10.497239 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:10.497207    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2fmw5_76bc7a14-41f9-4205-89c6-3a3a4f61cbe5/discovery/0.log"
Sep 17 14:52:10.547490 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:10.547456    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-549869464b-7pvhb_73a95568-9bbe-47dc-a8ea-de9b6a5eff04/kube-auth-proxy/0.log"
Sep 17 14:52:10.709561 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:10.709511    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-q95h6_c9df4506-8383-4582-92b0-b8c06c843697/payload-pre-processing/0.log"
Sep 17 14:52:10.757065 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:10.756978    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fkwcw_f5f4a040-2239-41eb-b7c7-c941c7a3e30c/payload-processing/0.log"
Sep 17 14:52:11.852103 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:11.852065    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zsfks_916ac6c2-3233-45cd-9c79-088264c87d00/serve-healthcheck-canary/0.log"
Sep 17 14:52:12.507127 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:12.507098    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7v4c_6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5/kube-rbac-proxy/0.log"
Sep 17 14:52:12.525942 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:12.525898    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7v4c_6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5/exporter/0.log"
Sep 17 14:52:12.545741 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:12.545717    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7v4c_6a66b70e-4c3a-4a3c-b374-d60c2e06cdd5/extractor/0.log"
Sep 17 14:52:12.774983 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:12.774911    2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-6fbhq"
Sep 17 14:52:14.966124 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:14.966097    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-j4c8c_3164cd21-60ad-4914-ac6f-769b3489289e/manager/0.log"
Sep 17 14:52:15.329425 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:15.329352    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bf6d9dbbc-5jnd8_91c72e06-3799-48c4-b06d-eba49a8e997d/manager/0.log"
Sep 17 14:52:15.353560 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:15.353534    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ccvnw_98afef09-5ea8-49e4-b0fb-b11305bddd31/server/0.log"
Sep 17 14:52:15.497087 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:15.497049    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hxhh6_c47cc71d-8718-4874-a0ed-8f1fb3e53be0/manager/0.log"
Sep 17 14:52:15.663809 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:15.663780    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-569db487f9-q92qv_fe7032d6-ba74-471f-90e2-b539862c19ec/manager/0.log"
Sep 17 14:52:15.683700 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:15.683674    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-569db487f9-q92qv_fe7032d6-ba74-471f-90e2-b539862c19ec/copy-manifests/0.log"
Sep 17 14:52:16.913947 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:16.913906    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2fmw5_76bc7a14-41f9-4205-89c6-3a3a4f61cbe5/discovery/0.log"
Sep 17 14:52:16.923137 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:16.923112    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-549869464b-7pvhb_73a95568-9bbe-47dc-a8ea-de9b6a5eff04/kube-auth-proxy/0.log"
Sep 17 14:52:16.989670 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:16.989637    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-q95h6_c9df4506-8383-4582-92b0-b8c06c843697/payload-pre-processing/0.log"
Sep 17 14:52:17.022502 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:17.022464    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fkwcw_f5f4a040-2239-41eb-b7c7-c941c7a3e30c/payload-processing/0.log"
Sep 17 14:52:19.295259 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:19.295223    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2fmw5_76bc7a14-41f9-4205-89c6-3a3a4f61cbe5/discovery/0.log"
Sep 17 14:52:19.317286 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:19.317257    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-549869464b-7pvhb_73a95568-9bbe-47dc-a8ea-de9b6a5eff04/kube-auth-proxy/0.log"
Sep 17 14:52:19.399808 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:19.399764    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-q95h6_c9df4506-8383-4582-92b0-b8c06c843697/payload-pre-processing/0.log"
Sep 17 14:52:19.436723 ip-10-0-133-98 kubenswrapper[2537]: I0917 14:52:19.436690    2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fkwcw_f5f4a040-2239-41eb-b7c7-c941c7a3e30c/payload-processing/0.log"