Oct 30 14:06:03.004041 ip-10-0-129-158 systemd[1]: Starting Kubernetes Kubelet... Oct 30 14:06:03.426791 ip-10-0-129-158 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:06:03.426791 ip-10-0-129-158 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 14:06:03.426791 ip-10-0-129-158 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:06:03.426791 ip-10-0-129-158 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:06:03.434046 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.433960 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:03.434046 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434043 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:03.434046 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434047 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:03.434046 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434050 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434054 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434057 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434060 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434063 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434065 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434068 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434071 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434074 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434082 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434085 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434088 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434091 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434093 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434096 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434098 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434101 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434104 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434106 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:03.434163 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434109 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434112 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434115 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434117 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434120 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434122 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434125 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434127 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434129 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434132 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434135 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434137 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434140 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434143 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434145 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434148 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434152 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434155 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434158 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:03.434610 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434160 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434163 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434165 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434168 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434172 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434177 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434180 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434183 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434186 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434188 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434191 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434193 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434196 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434198 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434201 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434204 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434206 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434209 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434212 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:03.435072 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434214 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434217 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434219 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434222 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434224 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434227 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434230 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434233 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434236 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434240 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434243 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434246 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434249 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434251 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434254 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434256 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434259 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434261 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434264 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434266 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:03.435518 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434269 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434272 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434274 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434277 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434279 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434283 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434287 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434291 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434294 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434296 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434299 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434301 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434304 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434307 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434310 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434313 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434316 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434319 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434321 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:03.436005 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434325 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434327 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434330 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434334 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434337 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434340 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.434342 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435211 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435219 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435223 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435227 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435230 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435234 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435237 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435240 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435243 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435246 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435249 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435251 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:03.436549 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435254 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435256 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435259 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435261 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435264 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435266 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435269 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435272 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435275 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435277 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435280 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435283 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435286 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435288 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435291 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435293 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435297 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435301 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435305 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:03.437056 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435308 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435311 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435314 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435316 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435319 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435322 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435324 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435327 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435329 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435332 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435334 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435337 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435339 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435342 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435344 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435347 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435350 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435352 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435355 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435357 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:03.437514 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435360 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435363 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435365 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435368 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435371 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435373 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435377 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435379 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435382 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435384 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435387 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435389 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435392 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435395 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435397 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435400 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435403 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435406 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435409 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:03.438002 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435411 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435414 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435416 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435419 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435421 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435424 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435426 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435429 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435431 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435434 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435437 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435439 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435442 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435445 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435447 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435450 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435452 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435455 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435458 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435461 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:03.438451 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435463 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435466 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435469 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435471 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435474 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435476 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435479 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435482 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435484 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435488 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435492 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435496 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435500 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435503 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435506 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.435509 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436060 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436067 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436071 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:03.438953 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436076 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436083 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436086 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436089 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436092 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436095 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436098 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436100 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436103 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436105 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436108 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436110 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436113 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436115 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436118 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436121 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436123 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436126 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436129 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436133 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:03.439407 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436135 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436138 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436140 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436143 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436146 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436148 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436151 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436153 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436156 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436159 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436161 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436164 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436167 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436170 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436173 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436176 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436178 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436181 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436183 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436186 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:03.439895 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436189 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436191 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436194 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436196 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436199 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436201 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436204 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436206 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436209 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436211 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436214 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436217 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436220 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436222 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436225 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436228 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436230 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436233 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436237 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:03.440370 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436239 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436242 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436245 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436247 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436250 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436253 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436257 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436260 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436263 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436266 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436270 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436274 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436278 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436280 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436283 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436286 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436288 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436290 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436293 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436296 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:03.440823 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436298 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436301 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436303 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436306 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436309 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436311 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436315 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436317 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436320 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436323 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436325 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436328 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436330 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436333 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436335 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436338 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436341 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436345 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436350 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:03.441314 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436353 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436356 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436358 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436361 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.436364 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436461 2584 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436468 2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436475 2584 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436482 2584 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436487 2584 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436491 2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436495 2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436499 2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436502 2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436505 2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436510 2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436513 2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436516 2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436519 2584 flags.go:64] FLAG: --cgroup-root="" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436522 2584 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436526 2584 flags.go:64] FLAG: --client-ca-file="" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436529 2584 flags.go:64] FLAG: --cloud-provider="external" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436532 2584 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 14:06:03.441849 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436537 2584 flags.go:64] FLAG: --cluster-domain="" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436540 2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436543 2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436546 2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436550 2584 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436556 2584 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436559 2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436563 2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436568 2584 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436571 2584 flags.go:64] FLAG: --contention-profiling="false" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436574 2584 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436578 2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436581 2584 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436584 2584 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436588 2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436592 2584 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436594 2584 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436597 2584 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436601 2584 flags.go:64] FLAG: --enable-server="true" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436604 2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436608 2584 flags.go:64] FLAG: --event-burst="100" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436611 2584 flags.go:64] FLAG: --event-qps="50" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436614 2584 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436618 2584 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 14:06:03.442414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436621 2584 flags.go:64] FLAG: --eviction-hard="" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436625 2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436628 2584 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436631 2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436634 2584 flags.go:64] FLAG: --eviction-soft="" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436637 2584 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436641 2584 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436644 2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436647 2584 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436650 2584 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436653 2584 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436656 2584 flags.go:64] FLAG: --feature-gates="" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436661 2584 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436665 2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436668 2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436671 2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436676 2584 flags.go:64] FLAG: --healthz-port="10248" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436679 2584 flags.go:64] FLAG: --help="false" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436682 2584 flags.go:64] FLAG: --hostname-override="ip-10-0-129-158.ec2.internal" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436686 2584 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436689 2584 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436692 2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436695 2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 14:06:03.443022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436698 2584 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436701 2584 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436704 2584 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436707 2584 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436710 2584 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436725 2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436728 2584 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436731 2584 flags.go:64] FLAG: --kube-reserved="" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436734 2584 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436738 2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436741 2584 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436743 2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436746 2584 flags.go:64] FLAG: --lock-file="" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436750 2584 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436752 2584 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436755 2584 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436761 2584 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436764 2584 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436767 2584 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436770 2584 flags.go:64] FLAG: --logging-format="text" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436773 2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436776 2584 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436780 2584 flags.go:64] FLAG: --manifest-url="" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436783 2584 flags.go:64] FLAG: --manifest-url-header="" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436788 2584 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 14:06:03.443599 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436794 2584 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436798 2584 flags.go:64] FLAG: --max-pods="110" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436801 2584 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436804 2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436808 2584 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436811 2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436814 2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436817 2584 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436820 2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436828 2584 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436831 2584 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436834 2584 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436837 2584 flags.go:64] FLAG: --pod-cidr="" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436840 2584 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436843 2584 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436846 2584 flags.go:64] FLAG: --pods-per-core="0" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436849 2584 flags.go:64] FLAG: --port="10250" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436852 2584 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436855 2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dc2660bf87d08fc7" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436858 2584 flags.go:64] FLAG: --qos-reserved="" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436861 2584 flags.go:64] FLAG: --read-only-port="10255" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436864 2584 flags.go:64] FLAG: --register-node="true" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436868 2584 flags.go:64] FLAG: --register-with-taints="" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436875 2584 flags.go:64] FLAG: --registry-burst="10" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436878 2584 flags.go:64] FLAG: --registry-qps="5" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436882 2584 flags.go:64] FLAG: --reserved-cpus="" Oct 30 14:06:03.444204 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436885 2584 flags.go:64] FLAG: --reserved-memory="" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436888 2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436891 2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436896 2584 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436899 2584 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436902 2584 flags.go:64] FLAG: --runonce="false" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436906 2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436909 2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436912 2584 flags.go:64] FLAG: --seccomp-default="false" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436916 2584 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436919 2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436922 2584 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436925 2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436928 2584 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436931 2584 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436934 2584 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436937 2584 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436940 2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436943 2584 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436947 2584 flags.go:64] FLAG: --system-cgroups="" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436950 2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436956 2584 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436959 2584 flags.go:64] FLAG: --tls-cert-file="" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436962 2584 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436967 2584 flags.go:64] FLAG: --tls-min-version="" Oct 30 14:06:03.445111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436970 2584 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436973 2584 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436976 2584 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436979 2584 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436982 2584 flags.go:64] FLAG: --v="2" Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436986 2584 flags.go:64] FLAG: --version="false" Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436992 2584 flags.go:64] FLAG: --vmodule="" Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436996 2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.436999 2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437138 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437144 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437147 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437150 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437154 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437156 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437159 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437161 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437164 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437166 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437170 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437172 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437175 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:03.445815 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437178 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437180 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437183 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437187 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437191 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437194 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437197 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437199 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437202 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437204 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437207 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437209 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437225 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437228 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437231 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437234 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437236 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437239 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437242 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:03.446339 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437245 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437248 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437251 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437254 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437257 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437259 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437262 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437265 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437267 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437270 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437272 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437275 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437278 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437281 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437283 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437286 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437289 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437291 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437294 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437296 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:03.446819 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437299 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437301 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437304 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437307 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437309 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437311 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437314 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437317 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437319 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437322 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437324 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437327 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437330 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437334 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437337 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437339 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437342 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437344 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437347 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437349 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:03.447303 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437352 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437354 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437357 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437359 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437362 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437364 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437367 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437370 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437374 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437377 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437380 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437382 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437385 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437388 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437390 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437393 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437396 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437398 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437400 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:03.447794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437403 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437405 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437408 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437410 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437413 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437416 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437420 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437423 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437425 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437428 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437430 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437433 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437435 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437438 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437441 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.437467 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:06:03.448254 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437520 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437525 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437529 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437532 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437535 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437538 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437541 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437544 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437546 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437549 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437551 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437554 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437557 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437559 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437562 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437564 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437567 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437569 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437572 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437574 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:03.448636 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437577 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437580 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437588 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437591 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437594 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437597 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437599 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437602 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437605 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437607 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437609 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437612 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437615 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437617 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437620 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437623 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437625 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437628 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437630 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437633 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:03.449109 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437635 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437638 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437640 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437643 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437645 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437648 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437650 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437653 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437657 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437660 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437662 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437665 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437668 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437670 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437676 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437683 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437686 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437689 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437691 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:03.449583 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437694 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437696 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437699 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437702 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437704 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437708 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437722 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437725 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437728 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437731 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437733 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437736 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437738 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437741 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437744 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437746 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437749 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437751 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437754 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437756 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:03.450167 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437759 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437761 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437764 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437766 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437769 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437772 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437774 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437779 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437782 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437785 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437787 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437791 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437795 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437798 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437801 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437804 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437807 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437810 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437812 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:03.450635 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437815 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437819 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437822 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437824 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437827 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437829 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437832 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.437834 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.446467 2584 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.446484 2584 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.446502 2584 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.446508 2584 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446568 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446574 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446577 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446580 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446583 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446586 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446589 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446592 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:03.451139 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446595 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446598 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446601 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446603 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446606 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446609 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446612 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446614 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446617 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446620 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446623 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446625 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446628 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446631 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446634 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446636 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446639 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446642 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446644 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446646 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:03.451606 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446650 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446653 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446665 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446669 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446671 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446674 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446677 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446680 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446682 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446686 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446688 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446691 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446693 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446696 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446699 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446701 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446704 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446706 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446709 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:03.452093 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446725 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446730 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446733 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446736 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446738 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446741 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446744 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446746 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446749 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446751 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446754 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446757 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446760 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446763 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446767 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446769 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446772 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446775 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446778 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446780 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:03.452531 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446783 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446785 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446788 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446790 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446793 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446795 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446798 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446801 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446803 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446806 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446809 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446811 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446815 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446817 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446821 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446823 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446826 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446829 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446831 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:03.453009 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446834 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446837 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446839 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446842 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446844 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446847 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446850 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446853 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446856 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446858 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446861 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446864 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446868 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446873 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446876 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446879 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446882 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446885 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446887 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:03.453442 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.446890 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.446917 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447063 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447069 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447073 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447076 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447079 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447082 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447085 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447087 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447090 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447093 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447096 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447098 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447101 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447104 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:03.453902 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447106 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447109 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447112 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447115 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447117 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447120 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447123 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447126 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447129 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447131 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447134 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447136 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447139 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447142 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447144 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447147 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447150 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447152 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447155 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447157 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:03.454272 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447160 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447162 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447165 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447167 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447171 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447173 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447176 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447178 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447181 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447185 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447190 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447193 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447195 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447198 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447201 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447204 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447206 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447209 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447212 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:03.454794 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447214 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447217 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447220 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447222 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447225 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447227 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447230 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447233 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447235 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447238 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447241 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447245 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447248 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447251 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447254 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447257 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447259 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447262 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447265 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:03.455244 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447270 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447274 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447279 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447283 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447288 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447292 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447296 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447299 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447301 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447304 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447307 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447310 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447313 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447316 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447318 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447321 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447324 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447327 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447329 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447332 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:03.455706 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447335 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447337 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447340 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447342 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447345 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447347 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447350 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447352 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447355 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447357 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447360 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447363 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447366 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447369 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.447392 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447427 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:03.456182 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447431 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447434 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447437 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447440 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447443 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447445 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447448 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447451 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447454 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447456 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447459 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447462 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447464 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447467 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447470 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447472 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447475 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447477 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447480 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447482 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:03.456562 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447485 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447490 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447493 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447496 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447498 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447501 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447503 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447506 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447509 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447512 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447514 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447517 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447520 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447524 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447528 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447531 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447534 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447537 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447540 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:03.457088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447543 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447545 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447548 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447550 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447553 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447556 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447559 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447561 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447564 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447567 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447569 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447572 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447574 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447577 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447581 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447585 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447588 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447592 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447594 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:03.457564 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447597 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447600 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447603 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447606 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447609 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447612 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447615 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447618 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447620 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447623 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447625 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447628 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447630 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447633 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447635 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447638 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447640 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447643 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447645 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447648 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:03.458030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447650 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447653 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447656 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447658 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447661 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447663 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447666 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447670 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447672 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447675 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447678 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447680 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447683 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447685 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447688 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447691 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447693 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447698 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447700 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447703 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:03.458506 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447706 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447708 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447711 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447726 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447730 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447733 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:03.447737 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.447793 2584 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.450191 2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.451272 2584 server.go:1014] "Starting client certificate rotation" Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.451368 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.451407 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.453591 2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:06:03.459060 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.455229 2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:06:03.467702 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.467686 2584 log.go:25] "Validated CRI v1 runtime API" Oct 30 14:06:03.472785 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.472770 2584 log.go:25] "Validated CRI v1 image API" Oct 30 14:06:03.473535 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.473512 2584 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 14:06:03.479585 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.479561 2584 fs.go:135] Filesystem UUIDs: map[0058fbd1-3288-49a7-b7be-227d2a6d2706:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f3886125-b35a-4731-a84d-2baf6ef0f019:/dev/nvme0n1p4] Oct 30 14:06:03.479656 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.479584 2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 14:06:03.485253 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.485117 2584 manager.go:217] Machine: {Timestamp:2026-10-30 14:06:03.483579472 +0000 UTC m=+0.368337729 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3130379 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21ff87d8ab27c55902169e3c643934 SystemUUID:ec21ff87-d8ab-27c5-5902-169e3c643934 BootID:8073ca32-e3f7-4345-aeb7-f19e25acf6c2 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d0:a1:ef:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d0:a1:ef:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:b4:ff:3d:22:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 14:06:03.485334 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.485254 2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 14:06:03.485379 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.485340 2584 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 14:06:03.487403 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.487374 2584 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 14:06:03.487545 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.487406 2584 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-158.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 14:06:03.487586 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.487548 2584 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 14:06:03.487586 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.487557 2584 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 14:06:03.487586 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.487582 2584 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:06:03.487667 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.487601 2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:06:03.488795 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.488784 2584 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 14:06:03.493859 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.493844 2584 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 14:06:03.493982 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.493972 2584 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 14:06:03.494938 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.494925 2584 kubelet.go:493] "Attempting to sync node with API server" Oct 30 14:06:03.494969 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.494945 2584 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 14:06:03.494969 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.494959 2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 14:06:03.495044 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.494973 2584 kubelet.go:405] "Adding apiserver pod source" Oct 30 14:06:03.495044 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.494985 2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 14:06:03.495386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.495372 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:06:03.498047 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.498029 2584 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 14:06:03.504573 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.504558 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:06:03.505528 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.505516 2584 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 14:06:03.505583 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.505546 2584 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 14:06:03.505744 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.505707 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 14:06:03.505744 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.505736 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 14:06:03.505744 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.505742 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 14:06:03.505838 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.505748 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 14:06:03.505838 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.505754 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 14:06:03.505838 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.505760 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 14:06:03.506509 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.506500 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 14:06:03.506540 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.506509 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 14:06:03.506540 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.506517 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 14:06:03.506540 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.506523 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 14:06:03.506540 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.506532 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 14:06:03.506540 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.506540 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 14:06:03.507313 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.507303 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 14:06:03.507313 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.507313 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 14:06:03.511180 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.511167 2584 server.go:1263] "Started kubelet" Oct 30 14:06:03.511266 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.511237 2584 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 14:06:03.511365 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.511317 2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 14:06:03.511407 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.511395 2584 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 14:06:03.513680 ip-10-0-129-158 systemd[1]: Started Kubernetes Kubelet. Oct 30 14:06:03.514039 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.513709 2584 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 14:06:03.515354 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.515308 2584 server.go:317] "Adding debug handlers to kubelet server" Oct 30 14:06:03.517052 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.517035 2584 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 14:06:03.520082 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.520065 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 14:06:03.520960 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.520830 2584 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 14:06:03.521104 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.521013 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:03.521622 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.521609 2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 14:06:03.521773 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.521612 2584 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 14:06:03.521773 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.521641 2584 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 14:06:03.521773 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.521731 2584 reconstruct.go:97] "Volume reconstruction finished" Oct 30 14:06:03.521773 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.521741 2584 reconciler.go:29] "Reconciler: start to sync state" Oct 30 14:06:03.521974 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.521793 2584 factory.go:55] Registering systemd factory Oct 30 14:06:03.521974 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.521808 2584 factory.go:223] Registration of the systemd container factory successfully Oct 30 14:06:03.522074 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.522028 2584 factory.go:153] Registering CRI-O factory Oct 30 14:06:03.522074 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.522039 2584 factory.go:223] Registration of the crio container factory successfully Oct 30 14:06:03.522164 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.522106 2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 14:06:03.522164 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.522124 2584 factory.go:103] Registering Raw factory Oct 30 14:06:03.522164 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.522133 2584 manager.go:1199] Started watching for new ooms in manager Oct 30 14:06:03.523364 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.523346 2584 manager.go:319] Starting recovery of all containers Oct 30 14:06:03.535638 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.535621 2584 manager.go:324] Recovery completed Oct 30 14:06:03.540530 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.540514 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:06:03.543691 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.543665 2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-158.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 14:06:03.544014 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.543951 2584 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 14:06:03.544081 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.544050 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 14:06:03.544990 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.543659 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-158.ec2.internal.18e353c0a638890c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-158.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-158.ec2.internal,},FirstTimestamp:2026-10-30 14:06:03.511138572 +0000 UTC m=+0.395896829,LastTimestamp:2026-10-30 14:06:03.511138572 +0000 UTC m=+0.395896829,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-158.ec2.internal,}" Oct 30 14:06:03.545414 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.545394 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:06:03.545504 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.545430 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:06:03.545504 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.545440 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:06:03.546036 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.546022 2584 cpu_manager.go:226] "Starting" policy="none" Oct 30 14:06:03.546036 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.546033 2584 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 14:06:03.546149 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.546047 2584 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 14:06:03.546149 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.546100 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:06:03.547647 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.547633 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:06:03.547728 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.547656 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:06:03.547728 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.547669 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:06:03.548319 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.548308 2584 policy_none.go:50] "Start" Oct 30 14:06:03.548356 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.548323 2584 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 14:06:03.548356 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.548332 2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 14:06:03.550432 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.550417 2584 policy_none.go:44] "Start" Oct 30 14:06:03.562219 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.562133 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-158.ec2.internal.18e353c0a84387ad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-158.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-158.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-158.ec2.internal,},FirstTimestamp:2026-10-30 14:06:03.545413549 +0000 UTC m=+0.430171805,LastTimestamp:2026-10-30 14:06:03.545413549 +0000 UTC m=+0.430171805,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-158.ec2.internal,}" Oct 30 14:06:03.562851 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.562825 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 14:06:03.562943 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.562828 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 14:06:03.562943 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.562825 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 14:06:03.576118 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.576092 2584 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 14:06:03.578441 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.578425 2584 manager.go:341] "Starting Device Plugin manager" Oct 30 14:06:03.578519 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.578470 2584 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 14:06:03.578519 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.578481 2584 server.go:87] "Starting device plugin registration server" Oct 30 14:06:03.578674 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.578663 2584 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 14:06:03.578706 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.578675 2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 14:06:03.578795 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.578771 2584 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 14:06:03.578888 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.578875 2584 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 14:06:03.578936 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.578889 2584 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 14:06:03.579408 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.579389 2584 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 14:06:03.579496 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.579434 2584 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:03.586729 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.586694 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-khdgn" Oct 30 14:06:03.587143 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.587077 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-158.ec2.internal.18e353c0a843db03 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-158.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-158.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-158.ec2.internal,},FirstTimestamp:2026-10-30 14:06:03.545434883 +0000 UTC m=+0.430193140,LastTimestamp:2026-10-30 14:06:03.545434883 +0000 UTC m=+0.430193140,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-158.ec2.internal,}" Oct 30 14:06:03.597778 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.597762 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-khdgn" Oct 30 14:06:03.603553 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.603496 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-158.ec2.internal.18e353c0a843fe56 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-158.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-158.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-158.ec2.internal,},FirstTimestamp:2026-10-30 14:06:03.545443926 +0000 UTC m=+0.430202182,LastTimestamp:2026-10-30 14:06:03.545443926 +0000 UTC m=+0.430202182,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-158.ec2.internal,}" Oct 30 14:06:03.679586 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.679537 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:06:03.680403 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.680387 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:06:03.680457 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.680421 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:06:03.680457 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.680434 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:06:03.680521 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.680462 2584 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-158.ec2.internal" Oct 30 14:06:03.701554 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.701537 2584 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-158.ec2.internal" Oct 30 14:06:03.701601 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.701564 2584 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-158.ec2.internal\": node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:03.750516 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.748101 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:03.774742 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.774703 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 14:06:03.775809 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.775790 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 14:06:03.775867 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.775819 2584 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 14:06:03.775867 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.775851 2584 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 14:06:03.775965 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.775891 2584 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 14:06:03.793317 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.793301 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:06:03.848799 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.848754 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:03.876217 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.876196 2584 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal"] Oct 30 14:06:03.876278 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.876248 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:06:03.877174 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.877157 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:06:03.877265 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.877192 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:06:03.877265 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.877206 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:06:03.879514 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.879498 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:06:03.880195 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.879985 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:03.880195 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.880047 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:06:03.881836 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.880678 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:06:03.881836 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.880711 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:06:03.881836 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.880742 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:06:03.881836 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.881062 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:06:03.881836 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.881095 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:06:03.881836 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.881110 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:06:03.882994 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.882976 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:03.883076 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.883005 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:06:03.883787 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.883770 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:06:03.883847 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.883804 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:06:03.883847 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.883819 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:06:03.912224 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.912203 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-158.ec2.internal\" not found" node="ip-10-0-129-158.ec2.internal" Oct 30 14:06:03.915166 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.915150 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-158.ec2.internal\" not found" node="ip-10-0-129-158.ec2.internal" Oct 30 14:06:03.924723 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.924690 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:03.924780 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.924730 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:03.924780 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:03.924751 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/274ce57e4715bf968542a8e94100b125-config\") pod \"kube-apiserver-proxy-ip-10-0-129-158.ec2.internal\" (UID: \"274ce57e4715bf968542a8e94100b125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:03.949812 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:03.949777 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:04.025208 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.025186 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:04.025297 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.025227 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:04.025297 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.025246 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/274ce57e4715bf968542a8e94100b125-config\") pod \"kube-apiserver-proxy-ip-10-0-129-158.ec2.internal\" (UID: \"274ce57e4715bf968542a8e94100b125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:04.025363 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.025300 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/274ce57e4715bf968542a8e94100b125-config\") pod \"kube-apiserver-proxy-ip-10-0-129-158.ec2.internal\" (UID: \"274ce57e4715bf968542a8e94100b125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:04.025363 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.025314 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:04.025363 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.025314 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:04.050746 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:04.050709 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:04.151221 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:04.151198 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:04.215452 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.215401 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:04.218532 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.218517 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:04.252038 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:04.252015 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:04.352483 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:04.352448 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:04.450919 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.450887 2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 14:06:04.451311 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.451012 2584 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 14:06:04.453013 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:04.453000 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:04.553400 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:04.553346 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:04.576388 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.576370 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:06:04.577756 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.577742 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:06:04.577835 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.577768 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:06:04.577835 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.577777 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:06:04.589879 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.589852 2584 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:06:04.602910 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.602890 2584 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:06:04.603867 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.603840 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 14:01:03 +0000 UTC" deadline="2028-06-28 21:49:58.41958251 +0000 UTC" Oct 30 14:06:04.603867 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.603862 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14575h43m53.815722347s" Oct 30 14:06:04.621214 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.621192 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:04.644911 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.644893 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:06:04.646136 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.646124 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:04.662863 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.662849 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:06:04.710088 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:04.710057 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d3226536491d82b487daab954fd946.slice/crio-0bc892c923ac74be04bf036a21cf1b28787d5c72dba173b2e407434e9be65521 WatchSource:0}: Error finding container 0bc892c923ac74be04bf036a21cf1b28787d5c72dba173b2e407434e9be65521: Status 404 returned error can't find the container with id 0bc892c923ac74be04bf036a21cf1b28787d5c72dba173b2e407434e9be65521 Oct 30 14:06:04.710441 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:04.710425 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274ce57e4715bf968542a8e94100b125.slice/crio-f4dca8d9b5a5f58da6d765ffd9051266f97f624e74a03bf7ec76b713465e78e2 WatchSource:0}: Error finding container f4dca8d9b5a5f58da6d765ffd9051266f97f624e74a03bf7ec76b713465e78e2: Status 404 returned error can't find the container with id f4dca8d9b5a5f58da6d765ffd9051266f97f624e74a03bf7ec76b713465e78e2 Oct 30 14:06:04.716093 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.716081 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:06:04.779100 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.779066 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" event={"ID":"51d3226536491d82b487daab954fd946","Type":"ContainerStarted","Data":"0bc892c923ac74be04bf036a21cf1b28787d5c72dba173b2e407434e9be65521"} Oct 30 14:06:04.779926 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:04.779907 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" event={"ID":"274ce57e4715bf968542a8e94100b125","Type":"ContainerStarted","Data":"f4dca8d9b5a5f58da6d765ffd9051266f97f624e74a03bf7ec76b713465e78e2"} Oct 30 14:06:05.057680 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.057655 2584 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:06:05.496165 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.496138 2584 apiserver.go:52] "Watching apiserver" Oct 30 14:06:05.520841 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.520815 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 14:06:05.536212 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.536185 2584 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 14:06:05.536590 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.536567 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-mssbd","openshift-multus/multus-fp57b","kube-system/konnectivity-agent-h6hnr","kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-clqhk","openshift-multus/network-metrics-daemon-rgjxr","openshift-network-operator/iptables-alerter-xf6z2","openshift-cluster-node-tuning-operator/tuned-crmmp","openshift-dns/node-resolver-qnfg6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal","openshift-network-diagnostics/network-check-target-stdbh","kube-system/global-pull-secret-syncer-vlmdd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb","openshift-image-registry/node-ca-vpkj5"] Oct 30 14:06:05.539309 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.539281 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:05.539402 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:05.539381 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:05.541546 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.541525 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.543582 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.543550 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:05.546278 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.546256 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.548269 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.548251 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.550668 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.550653 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:05.550764 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:05.550747 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:05.556491 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.553388 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:05.556491 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.556282 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.558506 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.558462 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 14:06:05.558556 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.558529 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mfmhk\"" Oct 30 14:06:05.558652 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.558639 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 14:06:05.558960 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.558944 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 14:06:05.559193 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.559177 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 14:06:05.559937 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.559920 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:05.563479 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.562410 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 14:06:05.563479 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.562453 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 14:06:05.563479 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.562548 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 14:06:05.563479 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.562625 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 14:06:05.563479 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.562647 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qkz6n\"" Oct 30 14:06:05.563479 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.562548 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c5nrs\"" Oct 30 14:06:05.563479 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.562748 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 14:06:05.563479 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.562848 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 14:06:05.563479 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.562937 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 14:06:05.563479 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.563006 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-94q7c\"" Oct 30 14:06:05.563479 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.563074 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 14:06:05.564188 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.564124 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 14:06:05.564188 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.564172 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:05.565070 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.565049 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 14:06:05.567157 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.567110 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 14:06:05.567337 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.567319 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:05.567536 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.567501 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.569441 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.569420 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mgc9p" Oct 30 14:06:05.569537 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.569526 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 14:06:05.570010 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.569989 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:05.570100 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:05.570041 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:05.581293 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.581264 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mgc9p" Oct 30 14:06:05.623594 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.623575 2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 14:06:05.633627 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.633602 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.633751 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.633635 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-netns\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.633751 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.633660 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-device-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.633751 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.633735 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.633913 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.633833 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-daemon-config\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.633913 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.633867 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-modprobe-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.633913 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.633894 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.634062 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.633948 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.634062 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634033 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-etc-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.634166 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634062 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-config\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.634166 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634086 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.634166 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634111 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-k8s-cni-cncf-io\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.634166 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634136 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/354da4d4-65ce-4f03-a9aa-f7840bf28711-host\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:05.634166 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634157 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-systemd-units\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.634371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634179 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-var-lib-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.634371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634203 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-sys-fs\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.634371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634218 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-bin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.634371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634232 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-var-lib-kubelet\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.634371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634246 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-kubelet\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.634371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634262 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-netns\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.634371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634284 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysconfig\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.634371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634308 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-run\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.634371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634322 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-dbus\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:05.634371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634337 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ead802d8-7ee2-4381-ad65-723736aabd12-agent-certs\") pod \"konnectivity-agent-h6hnr\" (UID: \"ead802d8-7ee2-4381-ad65-723736aabd12\") " pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:05.634371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634363 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-cni-binary-copy\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.634703 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634386 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-host\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.634703 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634411 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-tuned\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.634703 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634425 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-registration-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.634703 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634438 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovn-node-metrics-cert\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.634703 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634457 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-script-lib\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.634703 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634481 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjlh\" (UniqueName: \"kubernetes.io/projected/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-kube-api-access-9gjlh\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:05.634703 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634504 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a8ec17f-ff03-4af2-8885-1707bf84ec85-iptables-alerter-script\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:05.634703 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634529 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:05.634703 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634554 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgn77\" (UniqueName: \"kubernetes.io/projected/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-kube-api-access-cgn77\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.634703 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634664 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634731 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-env-overrides\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634759 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-kubelet\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634795 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78841e01-5bf7-4348-9802-09a81adeb88a-tmp-dir\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634814 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-conf-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634864 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78841e01-5bf7-4348-9802-09a81adeb88a-hosts-file\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634889 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjm94\" (UniqueName: \"kubernetes.io/projected/78841e01-5bf7-4348-9802-09a81adeb88a-kube-api-access-xjm94\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634904 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/354da4d4-65ce-4f03-a9aa-f7840bf28711-serviceca\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634932 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634952 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-bin\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634978 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.634998 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qdqj\" (UniqueName: \"kubernetes.io/projected/3a8ec17f-ff03-4af2-8885-1707bf84ec85-kube-api-access-8qdqj\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635037 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-multus-certs\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635053 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xhlw\" (UniqueName: \"kubernetes.io/projected/5b388277-9bf2-45be-b85d-2eafd1a1c543-kube-api-access-5xhlw\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635069 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-kubelet-config\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635084 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.635163 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635098 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-slash\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635116 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635137 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-cnibin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635161 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-multus\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635209 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmq8x\" (UniqueName: \"kubernetes.io/projected/4f07e86b-0018-4c02-a9c6-cc24bfecd068-kube-api-access-wmq8x\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635225 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcd7x\" (UniqueName: \"kubernetes.io/projected/354da4d4-65ce-4f03-a9aa-f7840bf28711-kube-api-access-bcd7x\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635240 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-system-cni-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635253 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-node-log\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635274 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-netd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635295 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-socket-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635310 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-socket-dir-parent\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635325 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-conf\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635347 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-lib-modules\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635361 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ead802d8-7ee2-4381-ad65-723736aabd12-konnectivity-ca\") pod \"konnectivity-agent-h6hnr\" (UID: \"ead802d8-7ee2-4381-ad65-723736aabd12\") " pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635379 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-systemd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635402 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-log-socket\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.635901 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635421 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a8ec17f-ff03-4af2-8885-1707bf84ec85-host-slash\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635445 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-kubernetes\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635472 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-ovn\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635498 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pwr6\" (UniqueName: \"kubernetes.io/projected/70ab068a-ed6b-4fae-8019-2e7c5873ab43-kube-api-access-9pwr6\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635568 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw84x\" (UniqueName: \"kubernetes.io/projected/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kube-api-access-zw84x\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635592 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-hostroot\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635607 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-etc-kubernetes\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635622 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-tmp\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635637 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cnibin\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635659 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635683 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635697 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-sys\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635712 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7vhn\" (UniqueName: \"kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn\") pod \"network-check-target-stdbh\" (UID: \"aea273bc-03a8-41ad-8ed0-c1eab9a23a45\") " pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635747 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-system-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.636571 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.635762 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-systemd\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.659488 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.659463 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:05.659627 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.659578 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 14:06:05.659852 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.659801 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 14:06:05.660157 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660025 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:05.660157 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660076 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mlbs5\"" Oct 30 14:06:05.660157 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660084 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 14:06:05.660157 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660121 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 14:06:05.660157 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660128 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 14:06:05.660157 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660135 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 14:06:05.660452 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660183 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 14:06:05.660452 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660317 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5kvlg\"" Oct 30 14:06:05.660452 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660335 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w5lxm\"" Oct 30 14:06:05.660452 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660388 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5t6gk\"" Oct 30 14:06:05.660452 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660405 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dqvfb\"" Oct 30 14:06:05.660452 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660435 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:05.660644 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.660482 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 14:06:05.736020 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.735990 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.736020 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736027 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-sys\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.736218 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736057 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vhn\" (UniqueName: \"kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn\") pod \"network-check-target-stdbh\" (UID: \"aea273bc-03a8-41ad-8ed0-c1eab9a23a45\") " pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:05.736218 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736082 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-system-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.736218 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736104 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-systemd\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.736218 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736128 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.736218 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736178 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-netns\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.736218 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736202 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-device-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.736463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736225 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.736463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736248 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-daemon-config\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.736463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736275 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-modprobe-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.736463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736299 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.736463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736324 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.736463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736349 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-etc-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.736463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736371 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-config\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.736463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736396 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.736463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736421 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-k8s-cni-cncf-io\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.736463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736445 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/354da4d4-65ce-4f03-a9aa-f7840bf28711-host\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736473 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-systemd-units\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736498 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-var-lib-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736522 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-sys-fs\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736545 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-bin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736570 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-var-lib-kubelet\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736592 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-kubelet\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736614 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-netns\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736636 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysconfig\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736657 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-run\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736683 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-dbus\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736707 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ead802d8-7ee2-4381-ad65-723736aabd12-agent-certs\") pod \"konnectivity-agent-h6hnr\" (UID: \"ead802d8-7ee2-4381-ad65-723736aabd12\") " pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736751 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-cni-binary-copy\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736775 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-host\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736798 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-tuned\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736823 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-registration-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736847 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovn-node-metrics-cert\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.736884 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736874 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-script-lib\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736906 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjlh\" (UniqueName: \"kubernetes.io/projected/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-kube-api-access-9gjlh\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736931 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a8ec17f-ff03-4af2-8885-1707bf84ec85-iptables-alerter-script\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736955 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.736980 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgn77\" (UniqueName: \"kubernetes.io/projected/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-kube-api-access-cgn77\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737009 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737033 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-env-overrides\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737056 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-kubelet\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737079 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78841e01-5bf7-4348-9802-09a81adeb88a-tmp-dir\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737104 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-conf-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737136 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78841e01-5bf7-4348-9802-09a81adeb88a-hosts-file\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737163 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjm94\" (UniqueName: \"kubernetes.io/projected/78841e01-5bf7-4348-9802-09a81adeb88a-kube-api-access-xjm94\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737187 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/354da4d4-65ce-4f03-a9aa-f7840bf28711-serviceca\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737211 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737237 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-bin\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737261 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737285 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qdqj\" (UniqueName: \"kubernetes.io/projected/3a8ec17f-ff03-4af2-8885-1707bf84ec85-kube-api-access-8qdqj\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:05.737420 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737313 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-multus-certs\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737338 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhlw\" (UniqueName: \"kubernetes.io/projected/5b388277-9bf2-45be-b85d-2eafd1a1c543-kube-api-access-5xhlw\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737363 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-kubelet-config\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737389 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737414 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-slash\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737439 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737467 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-cnibin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737491 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-multus\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737518 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmq8x\" (UniqueName: \"kubernetes.io/projected/4f07e86b-0018-4c02-a9c6-cc24bfecd068-kube-api-access-wmq8x\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737542 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcd7x\" (UniqueName: \"kubernetes.io/projected/354da4d4-65ce-4f03-a9aa-f7840bf28711-kube-api-access-bcd7x\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737569 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-system-cni-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737594 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-node-log\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737634 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-netd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737659 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-socket-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737685 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-socket-dir-parent\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737710 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-conf\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737751 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-lib-modules\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.738011 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737780 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ead802d8-7ee2-4381-ad65-723736aabd12-konnectivity-ca\") pod \"konnectivity-agent-h6hnr\" (UID: \"ead802d8-7ee2-4381-ad65-723736aabd12\") " pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737807 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-systemd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737831 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-log-socket\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737856 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a8ec17f-ff03-4af2-8885-1707bf84ec85-host-slash\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737880 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-kubernetes\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737906 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-ovn\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737962 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pwr6\" (UniqueName: \"kubernetes.io/projected/70ab068a-ed6b-4fae-8019-2e7c5873ab43-kube-api-access-9pwr6\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.737990 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw84x\" (UniqueName: \"kubernetes.io/projected/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kube-api-access-zw84x\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.738016 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-hostroot\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.738043 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-etc-kubernetes\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.738068 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-tmp\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.738094 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cnibin\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.738121 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.738231 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.738377 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.738602 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.738428 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-sys\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.739121 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.738809 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-system-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.739121 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.738865 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-systemd\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.739337 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-var-lib-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.739431 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.739490 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.739512 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-conf-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.739551 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysconfig\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.739571 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78841e01-5bf7-4348-9802-09a81adeb88a-hosts-file\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.739350 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-bin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:05.739852 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:05.739943 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret podName:f6ae2497-ab63-4964-940a-e870d9131ae3 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:06.239902319 +0000 UTC m=+3.124660583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret") pod "global-pull-secret-syncer-vlmdd" (UID: "f6ae2497-ab63-4964-940a-e870d9131ae3") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.739963 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-k8s-cni-cncf-io\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.739337 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-netns\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740016 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/354da4d4-65ce-4f03-a9aa-f7840bf28711-host\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740062 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/354da4d4-65ce-4f03-a9aa-f7840bf28711-serviceca\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740063 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-systemd-units\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740104 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-var-lib-kubelet\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740109 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740140 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-bin\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.742111 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740153 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-kubelet\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740176 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a8ec17f-ff03-4af2-8885-1707bf84ec85-iptables-alerter-script\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:05.740193 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740200 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:05.740227 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs podName:a14b5c29-8cac-42f8-a3b2-6bf636cc05a4 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:06.240216354 +0000 UTC m=+3.124974599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs") pod "network-metrics-daemon-rgjxr" (UID: "a14b5c29-8cac-42f8-a3b2-6bf636cc05a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740244 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-run\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740331 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78841e01-5bf7-4348-9802-09a81adeb88a-tmp-dir\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740381 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-multus-certs\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740384 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-dbus\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740412 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-script-lib\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740433 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-sys-fs\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740508 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-modprobe-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740522 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-kubelet-config\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740544 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-device-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740570 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-netns\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740587 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740601 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-env-overrides\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.742951 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740641 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-host\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740666 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-lib-modules\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740691 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-slash\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740699 2584 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740745 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740802 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-cnibin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740826 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-multus\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740876 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-daemon-config\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740922 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.740950 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741070 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-hostroot\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741129 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-etc-kubernetes\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741143 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-system-cni-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741186 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-node-log\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741220 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-netd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741258 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ead802d8-7ee2-4381-ad65-723736aabd12-konnectivity-ca\") pod \"konnectivity-agent-h6hnr\" (UID: \"ead802d8-7ee2-4381-ad65-723736aabd12\") " pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741310 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-systemd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741309 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-socket-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.743630 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741346 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-log-socket\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741348 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-socket-dir-parent\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741367 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-cni-binary-copy\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741384 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-kubernetes\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741414 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a8ec17f-ff03-4af2-8885-1707bf84ec85-host-slash\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741452 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-conf\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741465 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-ovn\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741507 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cnibin\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741557 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-etc-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741604 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-registration-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741732 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-kubelet\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.741813 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.744386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.742073 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-config\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.744948 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.744927 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-tmp\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.745398 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.745370 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ead802d8-7ee2-4381-ad65-723736aabd12-agent-certs\") pod \"konnectivity-agent-h6hnr\" (UID: \"ead802d8-7ee2-4381-ad65-723736aabd12\") " pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:05.746004 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.745956 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-tuned\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.746098 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.746071 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovn-node-metrics-cert\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.759433 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:05.759388 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:06:05.759433 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:05.759410 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:06:05.759433 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:05.759425 2584 projected.go:196] Error preparing data for projected volume kube-api-access-f7vhn for pod openshift-network-diagnostics/network-check-target-stdbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:05.759626 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:05.759488 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn podName:aea273bc-03a8-41ad-8ed0-c1eab9a23a45 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:06.259466293 +0000 UTC m=+3.144224560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f7vhn" (UniqueName: "kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn") pod "network-check-target-stdbh" (UID: "aea273bc-03a8-41ad-8ed0-c1eab9a23a45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:05.764274 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.764235 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjlh\" (UniqueName: \"kubernetes.io/projected/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-kube-api-access-9gjlh\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:05.764494 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.764468 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjm94\" (UniqueName: \"kubernetes.io/projected/78841e01-5bf7-4348-9802-09a81adeb88a-kube-api-access-xjm94\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:05.767148 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.767123 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pwr6\" (UniqueName: \"kubernetes.io/projected/70ab068a-ed6b-4fae-8019-2e7c5873ab43-kube-api-access-9pwr6\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.768553 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.768469 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmq8x\" (UniqueName: \"kubernetes.io/projected/4f07e86b-0018-4c02-a9c6-cc24bfecd068-kube-api-access-wmq8x\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.768769 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.768647 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgn77\" (UniqueName: \"kubernetes.io/projected/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-kube-api-access-cgn77\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.769225 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.769202 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw84x\" (UniqueName: \"kubernetes.io/projected/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kube-api-access-zw84x\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.770688 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.770667 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhlw\" (UniqueName: \"kubernetes.io/projected/5b388277-9bf2-45be-b85d-2eafd1a1c543-kube-api-access-5xhlw\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.771155 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.771137 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcd7x\" (UniqueName: \"kubernetes.io/projected/354da4d4-65ce-4f03-a9aa-f7840bf28711-kube-api-access-bcd7x\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:05.771251 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.771200 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qdqj\" (UniqueName: \"kubernetes.io/projected/3a8ec17f-ff03-4af2-8885-1707bf84ec85-kube-api-access-8qdqj\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:05.827007 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.826979 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:06:05.856767 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.856734 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fp57b" Oct 30 14:06:05.867162 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.867138 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:05.867505 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:05.867478 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b388277_9bf2_45be_b85d_2eafd1a1c543.slice/crio-10c25997c409f0b29d95aa9f4484707384b72e3296304e35c7599b29249c9a5d WatchSource:0}: Error finding container 10c25997c409f0b29d95aa9f4484707384b72e3296304e35c7599b29249c9a5d: Status 404 returned error can't find the container with id 10c25997c409f0b29d95aa9f4484707384b72e3296304e35c7599b29249c9a5d Oct 30 14:06:05.879420 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:05.879398 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead802d8_7ee2_4381_ad65_723736aabd12.slice/crio-ff67637528372b258ad2cc6dd8f09ed6aeedd40078fbacaf0659ec0c99e3f98e WatchSource:0}: Error finding container ff67637528372b258ad2cc6dd8f09ed6aeedd40078fbacaf0659ec0c99e3f98e: Status 404 returned error can't find the container with id ff67637528372b258ad2cc6dd8f09ed6aeedd40078fbacaf0659ec0c99e3f98e Oct 30 14:06:05.886981 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.886960 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:05.902454 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.902435 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:05.904617 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.904594 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:05.906985 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.906962 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:05.909440 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.909422 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:05.918049 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.918029 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:05.921092 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:05.921066 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e6ce71_7fb4_40ab_b14a_54457b97bd6a.slice/crio-a4d69a5c88e477db630b182c8faf58c2aa21014498416fa1d31968f939c475fd WatchSource:0}: Error finding container a4d69a5c88e477db630b182c8faf58c2aa21014498416fa1d31968f939c475fd: Status 404 returned error can't find the container with id a4d69a5c88e477db630b182c8faf58c2aa21014498416fa1d31968f939c475fd Oct 30 14:06:05.924474 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:05.924453 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78841e01_5bf7_4348_9802_09a81adeb88a.slice/crio-a3db897157965152baeddf9496ee5d0dc0e65ea61064629d8ae00d218ac98cb1 WatchSource:0}: Error finding container a3db897157965152baeddf9496ee5d0dc0e65ea61064629d8ae00d218ac98cb1: Status 404 returned error can't find the container with id a3db897157965152baeddf9496ee5d0dc0e65ea61064629d8ae00d218ac98cb1 Oct 30 14:06:05.928033 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:05.928012 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:05.928367 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:05.928299 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ab068a_ed6b_4fae_8019_2e7c5873ab43.slice/crio-3f40c0d7ae2c4ae218631ecef0226219b409c88f84c1cc8682c6fa9d2519ba61 WatchSource:0}: Error finding container 3f40c0d7ae2c4ae218631ecef0226219b409c88f84c1cc8682c6fa9d2519ba61: Status 404 returned error can't find the container with id 3f40c0d7ae2c4ae218631ecef0226219b409c88f84c1cc8682c6fa9d2519ba61 Oct 30 14:06:05.929299 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:05.929276 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8ec17f_ff03_4af2_8885_1707bf84ec85.slice/crio-1804ec947805fc1806e9314ec207a3ffd212b9a4bc457bbd48a2ca0b28d02c2a WatchSource:0}: Error finding container 1804ec947805fc1806e9314ec207a3ffd212b9a4bc457bbd48a2ca0b28d02c2a: Status 404 returned error can't find the container with id 1804ec947805fc1806e9314ec207a3ffd212b9a4bc457bbd48a2ca0b28d02c2a Oct 30 14:06:05.931512 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:05.930941 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f07e86b_0018_4c02_a9c6_cc24bfecd068.slice/crio-5e8b776cb0b4aff552604151d213b8be7ca192b8cc2a87121aa2515c2eaba64d WatchSource:0}: Error finding container 5e8b776cb0b4aff552604151d213b8be7ca192b8cc2a87121aa2515c2eaba64d: Status 404 returned error can't find the container with id 5e8b776cb0b4aff552604151d213b8be7ca192b8cc2a87121aa2515c2eaba64d Oct 30 14:06:05.934936 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:05.934913 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354da4d4_65ce_4f03_a9aa_f7840bf28711.slice/crio-534e25f2af392926f0949ee3bb6b3799603647967a4fc967fe0e01ae8014d0dc WatchSource:0}: Error finding container 534e25f2af392926f0949ee3bb6b3799603647967a4fc967fe0e01ae8014d0dc: Status 404 returned error can't find the container with id 534e25f2af392926f0949ee3bb6b3799603647967a4fc967fe0e01ae8014d0dc Oct 30 14:06:05.939379 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:05.939357 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf753bb3b_c84d_4bd4_8b74_8ec50bb93c41.slice/crio-75c66915c3d4f4e4d4c22877e594eb5c908adc64357a5dc45d1ff3dc30feff24 WatchSource:0}: Error finding container 75c66915c3d4f4e4d4c22877e594eb5c908adc64357a5dc45d1ff3dc30feff24: Status 404 returned error can't find the container with id 75c66915c3d4f4e4d4c22877e594eb5c908adc64357a5dc45d1ff3dc30feff24 Oct 30 14:06:06.242629 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.242592 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:06.242762 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.242649 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:06.242828 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:06.242765 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:06.242879 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:06.242836 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret podName:f6ae2497-ab63-4964-940a-e870d9131ae3 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:07.242816225 +0000 UTC m=+4.127574470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret") pod "global-pull-secret-syncer-vlmdd" (UID: "f6ae2497-ab63-4964-940a-e870d9131ae3") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:06.242879 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:06.242765 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:06.242879 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:06.242877 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs podName:a14b5c29-8cac-42f8-a3b2-6bf636cc05a4 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:07.242869654 +0000 UTC m=+4.127627897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs") pod "network-metrics-daemon-rgjxr" (UID: "a14b5c29-8cac-42f8-a3b2-6bf636cc05a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:06.343699 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.343672 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vhn\" (UniqueName: \"kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn\") pod \"network-check-target-stdbh\" (UID: \"aea273bc-03a8-41ad-8ed0-c1eab9a23a45\") " pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:06.343845 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:06.343827 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:06:06.343914 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:06.343850 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:06:06.343914 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:06.343864 2584 projected.go:196] Error preparing data for projected volume kube-api-access-f7vhn for pod openshift-network-diagnostics/network-check-target-stdbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:06.344021 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:06.343925 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn podName:aea273bc-03a8-41ad-8ed0-c1eab9a23a45 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:07.343907619 +0000 UTC m=+4.228665876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7vhn" (UniqueName: "kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn") pod "network-check-target-stdbh" (UID: "aea273bc-03a8-41ad-8ed0-c1eab9a23a45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:06.585760 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.585015 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:01:05 +0000 UTC" deadline="2028-08-09 04:32:51.88335817 +0000 UTC" Oct 30 14:06:06.585760 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.585271 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15566h26m45.298093695s" Oct 30 14:06:06.776342 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.776308 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:06.776596 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:06.776441 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:06.785609 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.785579 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h6hnr" event={"ID":"ead802d8-7ee2-4381-ad65-723736aabd12","Type":"ContainerStarted","Data":"ff67637528372b258ad2cc6dd8f09ed6aeedd40078fbacaf0659ec0c99e3f98e"} Oct 30 14:06:06.788518 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.788486 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fp57b" event={"ID":"5b388277-9bf2-45be-b85d-2eafd1a1c543","Type":"ContainerStarted","Data":"10c25997c409f0b29d95aa9f4484707384b72e3296304e35c7599b29249c9a5d"} Oct 30 14:06:06.797040 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.797012 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" event={"ID":"274ce57e4715bf968542a8e94100b125","Type":"ContainerStarted","Data":"029b3058822bef799ba07838fbefb92bcc78629ee7ebbc0a6821db25bb93dc5f"} Oct 30 14:06:06.804853 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.804827 2584 generic.go:356] "Generic (PLEG): container finished" podID="51d3226536491d82b487daab954fd946" containerID="f641e47ff4b7e70294a2c16788c914c68916a1a4e0f000a35969e948c14fe276" exitCode=0 Oct 30 14:06:06.804954 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.804913 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" event={"ID":"51d3226536491d82b487daab954fd946","Type":"ContainerDied","Data":"f641e47ff4b7e70294a2c16788c914c68916a1a4e0f000a35969e948c14fe276"} Oct 30 14:06:06.812264 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.812227 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-crmmp" event={"ID":"4f07e86b-0018-4c02-a9c6-cc24bfecd068","Type":"ContainerStarted","Data":"5e8b776cb0b4aff552604151d213b8be7ca192b8cc2a87121aa2515c2eaba64d"} Oct 30 14:06:06.822140 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.822116 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xf6z2" event={"ID":"3a8ec17f-ff03-4af2-8885-1707bf84ec85","Type":"ContainerStarted","Data":"1804ec947805fc1806e9314ec207a3ffd212b9a4bc457bbd48a2ca0b28d02c2a"} Oct 30 14:06:06.839127 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.839039 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" event={"ID":"70ab068a-ed6b-4fae-8019-2e7c5873ab43","Type":"ContainerStarted","Data":"3f40c0d7ae2c4ae218631ecef0226219b409c88f84c1cc8682c6fa9d2519ba61"} Oct 30 14:06:06.846019 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.845995 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mssbd" event={"ID":"81e6ce71-7fb4-40ab-b14a-54457b97bd6a","Type":"ContainerStarted","Data":"a4d69a5c88e477db630b182c8faf58c2aa21014498416fa1d31968f939c475fd"} Oct 30 14:06:06.849921 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.849896 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" event={"ID":"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41","Type":"ContainerStarted","Data":"75c66915c3d4f4e4d4c22877e594eb5c908adc64357a5dc45d1ff3dc30feff24"} Oct 30 14:06:06.862990 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.862922 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vpkj5" event={"ID":"354da4d4-65ce-4f03-a9aa-f7840bf28711","Type":"ContainerStarted","Data":"534e25f2af392926f0949ee3bb6b3799603647967a4fc967fe0e01ae8014d0dc"} Oct 30 14:06:06.872916 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.872865 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qnfg6" event={"ID":"78841e01-5bf7-4348-9802-09a81adeb88a","Type":"ContainerStarted","Data":"a3db897157965152baeddf9496ee5d0dc0e65ea61064629d8ae00d218ac98cb1"} Oct 30 14:06:06.880575 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:06.880526 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" podStartSLOduration=2.880511135 podStartE2EDuration="2.880511135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:06.839040366 +0000 UTC m=+3.723798640" watchObservedRunningTime="2026-10-30 14:06:06.880511135 +0000 UTC m=+3.765269402" Oct 30 14:06:07.252207 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:07.252122 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:07.252207 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:07.252174 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:07.252434 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:07.252304 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:07.252434 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:07.252364 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs podName:a14b5c29-8cac-42f8-a3b2-6bf636cc05a4 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:09.252347446 +0000 UTC m=+6.137105697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs") pod "network-metrics-daemon-rgjxr" (UID: "a14b5c29-8cac-42f8-a3b2-6bf636cc05a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:07.252789 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:07.252769 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:07.252854 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:07.252831 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret podName:f6ae2497-ab63-4964-940a-e870d9131ae3 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:09.252814165 +0000 UTC m=+6.137572414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret") pod "global-pull-secret-syncer-vlmdd" (UID: "f6ae2497-ab63-4964-940a-e870d9131ae3") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:07.352812 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:07.352778 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vhn\" (UniqueName: \"kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn\") pod \"network-check-target-stdbh\" (UID: \"aea273bc-03a8-41ad-8ed0-c1eab9a23a45\") " pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:07.352981 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:07.352940 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:06:07.352981 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:07.352958 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:06:07.352981 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:07.352971 2584 projected.go:196] Error preparing data for projected volume kube-api-access-f7vhn for pod openshift-network-diagnostics/network-check-target-stdbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:07.353164 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:07.353027 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn podName:aea273bc-03a8-41ad-8ed0-c1eab9a23a45 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:09.353007908 +0000 UTC m=+6.237766157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7vhn" (UniqueName: "kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn") pod "network-check-target-stdbh" (UID: "aea273bc-03a8-41ad-8ed0-c1eab9a23a45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:07.585813 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:07.585730 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:01:05 +0000 UTC" deadline="2028-04-19 13:02:48.999413181 +0000 UTC" Oct 30 14:06:07.585813 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:07.585808 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12886h56m41.413610701s" Oct 30 14:06:07.776932 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:07.776893 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:07.777154 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:07.777037 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:07.777436 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:07.777416 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:07.777533 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:07.777512 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:07.895553 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:07.895290 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" event={"ID":"51d3226536491d82b487daab954fd946","Type":"ContainerStarted","Data":"09bf5ad456fba4d82631471d20fec0fe03ffb0a449291b3c7ba6d566d1535435"} Oct 30 14:06:08.776655 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:08.776626 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:08.777053 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:08.776783 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:08.903928 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:08.903890 2584 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="858e634874a0b293aea01d173c794374317a2cdaaa7ff62072a6766b551b7723" exitCode=0 Oct 30 14:06:08.904979 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:08.904952 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mssbd" event={"ID":"81e6ce71-7fb4-40ab-b14a-54457b97bd6a","Type":"ContainerDied","Data":"858e634874a0b293aea01d173c794374317a2cdaaa7ff62072a6766b551b7723"} Oct 30 14:06:08.975104 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:08.974870 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" podStartSLOduration=4.974851007 podStartE2EDuration="4.974851007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:07.93504158 +0000 UTC m=+4.819799864" watchObservedRunningTime="2026-10-30 14:06:08.974851007 +0000 UTC m=+5.859609255" Oct 30 14:06:09.268446 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:09.268357 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:09.268446 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:09.268414 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:09.268646 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:09.268548 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:09.268646 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:09.268611 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs podName:a14b5c29-8cac-42f8-a3b2-6bf636cc05a4 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:13.268592721 +0000 UTC m=+10.153350982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs") pod "network-metrics-daemon-rgjxr" (UID: "a14b5c29-8cac-42f8-a3b2-6bf636cc05a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:09.269024 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:09.269000 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:09.269106 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:09.269056 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret podName:f6ae2497-ab63-4964-940a-e870d9131ae3 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:13.26904179 +0000 UTC m=+10.153800047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret") pod "global-pull-secret-syncer-vlmdd" (UID: "f6ae2497-ab63-4964-940a-e870d9131ae3") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:09.369702 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:09.369667 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vhn\" (UniqueName: \"kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn\") pod \"network-check-target-stdbh\" (UID: \"aea273bc-03a8-41ad-8ed0-c1eab9a23a45\") " pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:09.369933 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:09.369844 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:06:09.369933 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:09.369861 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:06:09.369933 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:09.369874 2584 projected.go:196] Error preparing data for projected volume kube-api-access-f7vhn for pod openshift-network-diagnostics/network-check-target-stdbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:09.369933 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:09.369932 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn podName:aea273bc-03a8-41ad-8ed0-c1eab9a23a45 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:13.369915353 +0000 UTC m=+10.254673613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7vhn" (UniqueName: "kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn") pod "network-check-target-stdbh" (UID: "aea273bc-03a8-41ad-8ed0-c1eab9a23a45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:09.782887 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:09.782862 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:09.783327 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:09.782991 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:09.783384 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:09.783353 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:09.783466 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:09.783446 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:10.777568 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:10.777541 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:10.777681 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:10.777662 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:10.910981 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:10.910914 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h6hnr" event={"ID":"ead802d8-7ee2-4381-ad65-723736aabd12","Type":"ContainerStarted","Data":"303f845951eec7550c24beaf50cbee4a21958d9e90da74e14a31440d8e6f902e"} Oct 30 14:06:11.780759 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:11.780732 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:11.780888 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:11.780867 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:11.780946 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:11.780920 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:11.781022 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:11.781003 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:11.916650 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:11.916590 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" event={"ID":"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41","Type":"ContainerStarted","Data":"475d7f25beeb664e8941314b21b3bc5ae28fe3fc54e8b671e753b618b0a9c79c"} Oct 30 14:06:11.921275 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:11.921248 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vpkj5" event={"ID":"354da4d4-65ce-4f03-a9aa-f7840bf28711","Type":"ContainerStarted","Data":"99c7a08ac0c6eac9808055ed27c3d57140430ee718e4583310cc9a16880ea903"} Oct 30 14:06:11.969061 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:11.969010 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h6hnr" podStartSLOduration=5.138616767 podStartE2EDuration="8.968996991s" totalImagesPullingTime="3.830380224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:10.955400101 +0000 UTC m=+7.840158367" watchObservedRunningTime="2026-10-30 14:06:11.968996991 +0000 UTC m=+8.853755257" Oct 30 14:06:12.581522 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:12.581491 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:12.582160 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:12.582124 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:12.623281 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:12.623223 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vpkj5" podStartSLOduration=4.791455914 podStartE2EDuration="9.62319961s" totalImagesPullingTime="4.831743696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:11.970091853 +0000 UTC m=+8.854850124" watchObservedRunningTime="2026-10-30 14:06:12.62319961 +0000 UTC m=+9.507957882" Oct 30 14:06:12.776990 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:12.776540 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:12.776990 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:12.776655 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:12.805188 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:12.805140 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:12.808036 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:12.807826 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:13.298269 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.297675 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:13.298269 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.297735 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:13.298269 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:13.297852 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:13.298269 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:13.297915 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs podName:a14b5c29-8cac-42f8-a3b2-6bf636cc05a4 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:21.297897328 +0000 UTC m=+18.182655574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs") pod "network-metrics-daemon-rgjxr" (UID: "a14b5c29-8cac-42f8-a3b2-6bf636cc05a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:13.298747 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:13.298324 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:13.298747 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:13.298380 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret podName:f6ae2497-ab63-4964-940a-e870d9131ae3 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:21.298364249 +0000 UTC m=+18.183122546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret") pod "global-pull-secret-syncer-vlmdd" (UID: "f6ae2497-ab63-4964-940a-e870d9131ae3") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:13.398426 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.398386 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vhn\" (UniqueName: \"kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn\") pod \"network-check-target-stdbh\" (UID: \"aea273bc-03a8-41ad-8ed0-c1eab9a23a45\") " pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:13.398594 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:13.398549 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:06:13.398594 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:13.398567 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:06:13.398594 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:13.398580 2584 projected.go:196] Error preparing data for projected volume kube-api-access-f7vhn for pod openshift-network-diagnostics/network-check-target-stdbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:13.398757 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:13.398636 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn podName:aea273bc-03a8-41ad-8ed0-c1eab9a23a45 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:21.39861898 +0000 UTC m=+18.283377225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7vhn" (UniqueName: "kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn") pod "network-check-target-stdbh" (UID: "aea273bc-03a8-41ad-8ed0-c1eab9a23a45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:13.475214 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.475176 2584 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 14:06:13.587864 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.587773 2584 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T14:06:13.475193454Z","UUID":"a296ca7d-3463-43e6-9be2-0c344a5335cd","Handler":null,"Name":"","Endpoint":""} Oct 30 14:06:13.589955 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.589933 2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 14:06:13.590055 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.589964 2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 14:06:13.777804 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.777769 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:13.777974 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:13.777905 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:13.778439 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.778413 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:13.778582 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:13.778531 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:13.928063 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.928027 2584 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="26cb673bf2e93b36ad2a67f2bcfc95c778a59a25755c5273c6721f5a54832654" exitCode=0 Oct 30 14:06:13.928198 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.928112 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mssbd" event={"ID":"81e6ce71-7fb4-40ab-b14a-54457b97bd6a","Type":"ContainerDied","Data":"26cb673bf2e93b36ad2a67f2bcfc95c778a59a25755c5273c6721f5a54832654"} Oct 30 14:06:13.930157 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.930134 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" event={"ID":"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41","Type":"ContainerStarted","Data":"4369a7fc78c4db6d38cb23a8ec0bb7a1d5e17a8c133fcfcbd60932a99bb2e2d8"} Oct 30 14:06:13.932070 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.932046 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qnfg6" event={"ID":"78841e01-5bf7-4348-9802-09a81adeb88a","Type":"ContainerStarted","Data":"b31fc6d3dc92c4d73b7c549b7f517acf0ea413630744ffceca98ce1ce222fe81"} Oct 30 14:06:13.934282 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:13.934244 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fp57b" event={"ID":"5b388277-9bf2-45be-b85d-2eafd1a1c543","Type":"ContainerStarted","Data":"e61c575bfe94fafdfb23d877a8f4711a867d735ea8b2894c6c957f80de858716"} Oct 30 14:06:14.062447 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:14.062389 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-fp57b" podStartSLOduration=3.364131519 podStartE2EDuration="11.062369899s" totalImagesPullingTime="7.69823838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:14.022680082 +0000 UTC m=+10.907438347" watchObservedRunningTime="2026-10-30 14:06:14.062369899 +0000 UTC m=+10.947128200" Oct 30 14:06:14.777060 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:14.776821 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:14.777492 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:14.777180 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:14.939709 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:14.939194 2584 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="88f45a489cd1e0d10ccd3ce1019295c3cc367ac15e69f2b88cbb753e60e52cf8" exitCode=0 Oct 30 14:06:14.939709 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:14.939450 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mssbd" event={"ID":"81e6ce71-7fb4-40ab-b14a-54457b97bd6a","Type":"ContainerDied","Data":"88f45a489cd1e0d10ccd3ce1019295c3cc367ac15e69f2b88cbb753e60e52cf8"} Oct 30 14:06:14.942146 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:14.942054 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" event={"ID":"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41","Type":"ContainerStarted","Data":"af37f8455de5614c1fd5273a1a637b5b31f8c591e956f24bb2d5c89e674f78c1"} Oct 30 14:06:14.987140 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:14.986765 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qnfg6" podStartSLOduration=4.216195661 podStartE2EDuration="11.986746333s" totalImagesPullingTime="7.770550672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:14.063309698 +0000 UTC m=+10.948067965" watchObservedRunningTime="2026-10-30 14:06:14.986746333 +0000 UTC m=+11.871504599" Oct 30 14:06:15.029541 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:15.029499 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" podStartSLOduration=3.762012587 podStartE2EDuration="12.029485688s" totalImagesPullingTime="8.267473101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:06:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:15.028801289 +0000 UTC m=+11.913559556" watchObservedRunningTime="2026-10-30 14:06:15.029485688 +0000 UTC m=+11.914243956" Oct 30 14:06:15.776974 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:15.776936 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:15.777189 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:15.777066 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:15.777189 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:15.777120 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:15.777620 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:15.777229 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:15.946096 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:15.946047 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xf6z2" event={"ID":"3a8ec17f-ff03-4af2-8885-1707bf84ec85","Type":"ContainerStarted","Data":"220cd62fc13e0f8d1f292d98d099c7b2bb761987c277c3358a11cad880761739"} Oct 30 14:06:15.987051 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:15.987001 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xf6z2" podStartSLOduration=5.339764104 podStartE2EDuration="12.986986154s" totalImagesPullingTime="7.64722205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:15.986473547 +0000 UTC m=+12.871231813" watchObservedRunningTime="2026-10-30 14:06:15.986986154 +0000 UTC m=+12.871744761" Oct 30 14:06:16.776960 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:16.776930 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:16.777117 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:16.777047 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:16.951025 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:16.950990 2584 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="0791491ef5384b4d7ea36686a4d18725b019c51e560c5007b48b4236f71d6991" exitCode=0 Oct 30 14:06:16.951660 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:16.951074 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mssbd" event={"ID":"81e6ce71-7fb4-40ab-b14a-54457b97bd6a","Type":"ContainerDied","Data":"0791491ef5384b4d7ea36686a4d18725b019c51e560c5007b48b4236f71d6991"} Oct 30 14:06:16.952758 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:16.952732 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-crmmp" event={"ID":"4f07e86b-0018-4c02-a9c6-cc24bfecd068","Type":"ContainerStarted","Data":"04e15f1e5fd70c779b7d367c98d5c6e330c75780349632b1d5fb930ec61885ec"} Oct 30 14:06:17.065883 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:17.065788 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-crmmp" podStartSLOduration=3.931427966 podStartE2EDuration="14.065772149s" totalImagesPullingTime="10.134344183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:17.065331178 +0000 UTC m=+13.950089447" watchObservedRunningTime="2026-10-30 14:06:17.065772149 +0000 UTC m=+13.950530415" Oct 30 14:06:17.777095 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:17.777062 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:17.777285 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:17.777238 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:17.777285 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:17.777244 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:17.777421 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:17.777390 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:18.776696 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:18.776622 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:18.777271 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:18.776763 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:18.961340 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:18.961130 2584 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="3c46a6f54ad64892eeae35d566190b1b6e2a39343d551fc01bb1ca11fece5f02" exitCode=0 Oct 30 14:06:18.961340 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:18.961215 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mssbd" event={"ID":"81e6ce71-7fb4-40ab-b14a-54457b97bd6a","Type":"ContainerDied","Data":"3c46a6f54ad64892eeae35d566190b1b6e2a39343d551fc01bb1ca11fece5f02"} Oct 30 14:06:19.776597 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:19.776571 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:19.776772 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:19.776668 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:19.777109 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:19.776767 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:19.777109 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:19.776897 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:19.966578 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:19.966539 2584 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="779fac208966284b3d37094680e967633749bd27edc478c47043643dfbbaeb50" exitCode=0 Oct 30 14:06:19.966746 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:19.966585 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mssbd" event={"ID":"81e6ce71-7fb4-40ab-b14a-54457b97bd6a","Type":"ContainerDied","Data":"779fac208966284b3d37094680e967633749bd27edc478c47043643dfbbaeb50"} Oct 30 14:06:20.776180 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:20.776157 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:20.776279 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:20.776256 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:20.972411 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:20.972219 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" event={"ID":"70ab068a-ed6b-4fae-8019-2e7c5873ab43","Type":"ContainerStarted","Data":"feb504d0d9c2ca743ceb5539418bd08644134ee642de472743f0be6db5e23f7d"} Oct 30 14:06:20.972854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:20.972453 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" event={"ID":"70ab068a-ed6b-4fae-8019-2e7c5873ab43","Type":"ContainerStarted","Data":"3bdbe14238c63dbdd1ec3e642b487c4f4a49b8260deff92e20f7364a222843c3"} Oct 30 14:06:20.972854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:20.972485 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" event={"ID":"70ab068a-ed6b-4fae-8019-2e7c5873ab43","Type":"ContainerStarted","Data":"2166220011ced2e6a8d7d356991e4f4e1e95725b6314403692f7ad04a1cdd8bf"} Oct 30 14:06:20.972854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:20.972521 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" event={"ID":"70ab068a-ed6b-4fae-8019-2e7c5873ab43","Type":"ContainerStarted","Data":"f0aba1c49e5d6eb1a1d5fba461615a69be388e8393992022cf803c6bf5f4b7fc"} Oct 30 14:06:20.972854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:20.972553 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" event={"ID":"70ab068a-ed6b-4fae-8019-2e7c5873ab43","Type":"ContainerStarted","Data":"422e418b2a2524d13c3944a2a6c15c01dbe8d9f70320398532c6f2338e70e8f9"} Oct 30 14:06:20.972854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:20.972594 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" event={"ID":"70ab068a-ed6b-4fae-8019-2e7c5873ab43","Type":"ContainerStarted","Data":"a8fcc7f92c32362fabd7ca41fa72a6f015a93e646a4b33025034873784273ca4"} Oct 30 14:06:20.975470 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:20.975441 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mssbd" event={"ID":"81e6ce71-7fb4-40ab-b14a-54457b97bd6a","Type":"ContainerStarted","Data":"a6fa97fb29bfced3ee2794a86d9a34e547dc1eebb44bdc5bc04ad58710e0ddc8"} Oct 30 14:06:21.022816 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:21.022707 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mssbd" podStartSLOduration=7.368862102 podStartE2EDuration="18.022695981s" totalImagesPullingTime="10.653833879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:06:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:21.022076191 +0000 UTC m=+17.906834456" watchObservedRunningTime="2026-10-30 14:06:21.022695981 +0000 UTC m=+17.907454247" Oct 30 14:06:21.354342 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:21.354310 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:21.354342 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:21.354345 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:21.354523 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:21.354424 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:21.354523 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:21.354439 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:21.354523 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:21.354470 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs podName:a14b5c29-8cac-42f8-a3b2-6bf636cc05a4 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:37.35445767 +0000 UTC m=+34.239215927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs") pod "network-metrics-daemon-rgjxr" (UID: "a14b5c29-8cac-42f8-a3b2-6bf636cc05a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:06:21.354523 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:21.354482 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret podName:f6ae2497-ab63-4964-940a-e870d9131ae3 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:37.354477033 +0000 UTC m=+34.239235277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret") pod "global-pull-secret-syncer-vlmdd" (UID: "f6ae2497-ab63-4964-940a-e870d9131ae3") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:06:21.454985 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:21.454954 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vhn\" (UniqueName: \"kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn\") pod \"network-check-target-stdbh\" (UID: \"aea273bc-03a8-41ad-8ed0-c1eab9a23a45\") " pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:21.455119 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:21.455103 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:06:21.455152 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:21.455123 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:06:21.455152 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:21.455132 2584 projected.go:196] Error preparing data for projected volume kube-api-access-f7vhn for pod openshift-network-diagnostics/network-check-target-stdbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:21.455208 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:21.455176 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn podName:aea273bc-03a8-41ad-8ed0-c1eab9a23a45 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:37.455164445 +0000 UTC m=+34.339922690 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7vhn" (UniqueName: "kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn") pod "network-check-target-stdbh" (UID: "aea273bc-03a8-41ad-8ed0-c1eab9a23a45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:06:21.776470 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:21.776396 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:21.776612 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:21.776527 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:21.776612 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:21.776567 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:21.776684 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:21.776642 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:22.776866 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:22.776838 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:22.777242 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:22.776931 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:22.982750 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:22.982708 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" event={"ID":"70ab068a-ed6b-4fae-8019-2e7c5873ab43","Type":"ContainerStarted","Data":"fe5b957e5fabb3b179d44e0dc86a8bdb19741d95ed92a74ed3c13385355ce3e7"} Oct 30 14:06:23.777616 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:23.777586 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:23.778015 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:23.777735 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:23.778015 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:23.777764 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:23.778015 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:23.777821 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:24.776647 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:24.776617 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:24.776840 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:24.776725 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:25.776707 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:25.776676 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:25.777381 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:25.776734 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:25.777381 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:25.776822 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:25.777381 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:25.776942 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:25.992633 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:25.992596 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" event={"ID":"70ab068a-ed6b-4fae-8019-2e7c5873ab43","Type":"ContainerStarted","Data":"e591cab089dbaba64f4c9b7524f3c92bff0e7e835af614dd36ffae81b4769d05"} Oct 30 14:06:25.992872 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:25.992852 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:25.992963 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:25.992936 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:26.007752 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:26.007732 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:26.082063 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:26.082025 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" podStartSLOduration=8.50222585 podStartE2EDuration="23.082013326s" totalImagesPullingTime="14.579787476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:26.081898719 +0000 UTC m=+22.966656987" watchObservedRunningTime="2026-10-30 14:06:26.082013326 +0000 UTC m=+22.966771588" Oct 30 14:06:26.776821 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:26.776788 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:26.777289 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:26.776883 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:26.995409 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:26.995378 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:27.010790 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:27.010758 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:27.043644 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:27.043578 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vlmdd"] Oct 30 14:06:27.043776 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:27.043681 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:27.043830 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:27.043778 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:27.044201 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:27.044187 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-stdbh"] Oct 30 14:06:27.044260 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:27.044249 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:27.044334 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:27.044319 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:27.044726 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:27.044696 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rgjxr"] Oct 30 14:06:27.044817 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:27.044805 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:27.044899 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:27.044882 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:28.776189 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:28.776161 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:28.776772 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:28.776270 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:28.776772 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:28.776286 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:28.776772 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:28.776300 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:28.776772 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:28.776374 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:28.776772 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:28.776431 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:30.776614 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:30.776438 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:30.777031 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:30.776492 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:30.777031 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:30.776501 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:30.777031 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:30.776799 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:30.777031 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:30.776837 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:30.777031 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:30.776681 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:32.776668 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:32.776634 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:32.776668 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:32.776661 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:32.777095 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:32.776634 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:32.777095 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:32.776764 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgjxr" podUID="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" Oct 30 14:06:32.777095 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:32.776889 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stdbh" podUID="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" Oct 30 14:06:32.777095 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:32.776967 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlmdd" podUID="f6ae2497-ab63-4964-940a-e870d9131ae3" Oct 30 14:06:33.363573 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.363550 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeReady" Oct 30 14:06:33.363707 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.363663 2584 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 14:06:33.414607 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.414580 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jmrnq"] Oct 30 14:06:33.465683 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.465655 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dmj28"] Oct 30 14:06:33.465855 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.465794 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jmrnq" Oct 30 14:06:33.480995 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.480958 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm"] Oct 30 14:06:33.481962 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.481938 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 14:06:33.482109 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.481966 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dmj28" Oct 30 14:06:33.483386 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.482648 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 14:06:33.500916 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.500894 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f298s"] Oct 30 14:06:33.501009 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.500982 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:33.516093 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.516074 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 14:06:33.516501 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.516486 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tfvs9"] Oct 30 14:06:33.516603 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.516590 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f298s" Oct 30 14:06:33.530575 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.530554 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.530575 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.530554 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-84mcb"] Oct 30 14:06:33.549109 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.549087 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbcd9\" (UniqueName: \"kubernetes.io/projected/1cdffa3f-7378-4f50-b1bc-e87aa647c4d8-kube-api-access-tbcd9\") pod \"kas-connection-checker-685895c7c9-jmrnq\" (UID: \"1cdffa3f-7378-4f50-b1bc-e87aa647c4d8\") " pod="kube-system/kas-connection-checker-685895c7c9-jmrnq" Oct 30 14:06:33.549198 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.549138 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7rf\" (UniqueName: \"kubernetes.io/projected/3829121e-3790-47ff-b827-8774a70fec04-kube-api-access-7w7rf\") pod \"kas-connection-checker-685895c7c9-dmj28\" (UID: \"3829121e-3790-47ff-b827-8774a70fec04\") " pod="kube-system/kas-connection-checker-685895c7c9-dmj28" Oct 30 14:06:33.552844 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.552827 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-598ddffcff-nctk8"] Oct 30 14:06:33.552949 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.552935 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.576915 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.576889 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh"] Oct 30 14:06:33.577019 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.577005 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.585579 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.585562 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:33.585867 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.585847 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 14:06:33.585953 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.585851 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 14:06:33.585953 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.585851 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 14:06:33.586083 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.586064 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 14:06:33.598890 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.598871 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 14:06:33.599001 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.598902 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:33.599149 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.599133 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 14:06:33.599527 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.599511 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 14:06:33.602782 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.599997 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 14:06:33.602782 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.601567 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz"] Oct 30 14:06:33.602919 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.602830 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:33.603419 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.603402 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 14:06:33.616463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.616415 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 14:06:33.616545 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.616468 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 14:06:33.616545 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.616483 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 14:06:33.616545 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.616499 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 14:06:33.616545 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.616474 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 14:06:33.617147 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.617134 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 14:06:33.618645 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.618631 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8fff9cc46-btf9d"] Oct 30 14:06:33.618797 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.618783 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:33.633043 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.633022 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 14:06:33.640453 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.640436 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p"] Oct 30 14:06:33.640540 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.640530 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.649848 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.649831 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-stats-auth\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.649961 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.649857 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.649961 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.649876 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:33.649961 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.649893 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:33.650096 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.649960 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.650096 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.649996 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-snapshots\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.650096 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650078 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56lth\" (UniqueName: \"kubernetes.io/projected/a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5-kube-api-access-56lth\") pod \"kas-connection-checker-685895c7c9-f298s\" (UID: \"a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5\") " pod="kube-system/kas-connection-checker-685895c7c9-f298s" Oct 30 14:06:33.650240 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650105 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-default-certificate\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.650240 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650153 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-trusted-ca\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.650240 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650214 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7rf\" (UniqueName: \"kubernetes.io/projected/3829121e-3790-47ff-b827-8774a70fec04-kube-api-access-7w7rf\") pod \"kas-connection-checker-685895c7c9-dmj28\" (UID: \"3829121e-3790-47ff-b827-8774a70fec04\") " pod="kube-system/kas-connection-checker-685895c7c9-dmj28" Oct 30 14:06:33.650373 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650351 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-tmp\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.650429 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650414 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77kg8\" (UniqueName: \"kubernetes.io/projected/dfc6f25a-6a7a-472c-af78-683613ffb7a5-kube-api-access-77kg8\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.650484 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650429 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4171ff7a-5894-49d1-8683-606723fccdf9-serving-cert\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.650484 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650445 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sqjv\" (UniqueName: \"kubernetes.io/projected/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-kube-api-access-7sqjv\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:33.650484 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650464 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2zxn\" (UniqueName: \"kubernetes.io/projected/66b11a57-8746-4a93-b726-a877c2a22a98-kube-api-access-l2zxn\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.650589 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650573 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-config\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.650620 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650611 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-service-ca-bundle\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.650704 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650690 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.650770 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650748 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbcd9\" (UniqueName: \"kubernetes.io/projected/1cdffa3f-7378-4f50-b1bc-e87aa647c4d8-kube-api-access-tbcd9\") pod \"kas-connection-checker-685895c7c9-jmrnq\" (UID: \"1cdffa3f-7378-4f50-b1bc-e87aa647c4d8\") " pod="kube-system/kas-connection-checker-685895c7c9-jmrnq" Oct 30 14:06:33.650820 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650771 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssmtf\" (UniqueName: \"kubernetes.io/projected/4171ff7a-5894-49d1-8683-606723fccdf9-kube-api-access-ssmtf\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.650820 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.650803 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66b11a57-8746-4a93-b726-a877c2a22a98-serving-cert\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.654862 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.654796 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947"] Oct 30 14:06:33.654862 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.654859 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:33.674353 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.674322 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7rf\" (UniqueName: \"kubernetes.io/projected/3829121e-3790-47ff-b827-8774a70fec04-kube-api-access-7w7rf\") pod \"kas-connection-checker-685895c7c9-dmj28\" (UID: \"3829121e-3790-47ff-b827-8774a70fec04\") " pod="kube-system/kas-connection-checker-685895c7c9-dmj28" Oct 30 14:06:33.674353 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.674330 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbcd9\" (UniqueName: \"kubernetes.io/projected/1cdffa3f-7378-4f50-b1bc-e87aa647c4d8-kube-api-access-tbcd9\") pod \"kas-connection-checker-685895c7c9-jmrnq\" (UID: \"1cdffa3f-7378-4f50-b1bc-e87aa647c4d8\") " pod="kube-system/kas-connection-checker-685895c7c9-jmrnq" Oct 30 14:06:33.674950 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.674935 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-w9s2h"] Oct 30 14:06:33.675062 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.675046 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947" Oct 30 14:06:33.682809 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.682788 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 14:06:33.682908 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.682847 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:33.682908 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.682867 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 14:06:33.682908 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.682882 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:33.682908 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.682891 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:33.682908 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.682884 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:33.682908 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.682886 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 14:06:33.683160 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.682918 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 14:06:33.683160 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.682914 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 14:06:33.683160 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.682940 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 14:06:33.683160 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.683063 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:33.683160 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.683070 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:33.683744 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.683711 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 14:06:33.689076 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.689060 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-89594b78c-qxvnh"] Oct 30 14:06:33.689187 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.689173 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:33.699552 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.699534 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.699651 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.699540 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4"] Oct 30 14:06:33.714287 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.714265 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fvkvc"] Oct 30 14:06:33.714379 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.714367 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4" Oct 30 14:06:33.716963 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.716942 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 14:06:33.717055 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.716975 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 14:06:33.717269 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.717253 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vfwlz\"" Oct 30 14:06:33.717344 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.717280 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 14:06:33.717344 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.717291 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 14:06:33.717417 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.717386 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9wxhp\"" Oct 30 14:06:33.720619 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.717804 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 14:06:33.728570 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.728550 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m8jt7"] Oct 30 14:06:33.728707 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.728688 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.736119 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736102 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jmrnq"] Oct 30 14:06:33.736194 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736132 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dmj28"] Oct 30 14:06:33.736194 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736147 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f298s"] Oct 30 14:06:33.736194 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736167 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tfvs9"] Oct 30 14:06:33.736194 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736181 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-84mcb"] Oct 30 14:06:33.736194 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736195 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh"] Oct 30 14:06:33.736333 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736208 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-598ddffcff-nctk8"] Oct 30 14:06:33.736333 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736221 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8fff9cc46-btf9d"] Oct 30 14:06:33.736333 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736234 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947"] Oct 30 14:06:33.736333 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736233 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:33.736333 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736246 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-w9s2h"] Oct 30 14:06:33.736333 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736259 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fvkvc"] Oct 30 14:06:33.736333 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736280 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm"] Oct 30 14:06:33.736333 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736293 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-89594b78c-qxvnh"] Oct 30 14:06:33.736333 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736307 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz"] Oct 30 14:06:33.736333 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736319 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p"] Oct 30 14:06:33.736333 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736335 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4"] Oct 30 14:06:33.736709 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.736350 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m8jt7"] Oct 30 14:06:33.751574 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751543 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:33.751687 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751595 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-trusted-ca\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.751752 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751727 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thjb5\" (UniqueName: \"kubernetes.io/projected/4070c760-6728-4166-a333-77af48851fd8-kube-api-access-thjb5\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:33.751785 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751772 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66b11a57-8746-4a93-b726-a877c2a22a98-serving-cert\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.751835 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751811 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-snapshots\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.751885 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751834 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-service-ca-bundle\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.751885 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751852 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.751885 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751869 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:33.752019 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751891 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f62nk\" (UniqueName: \"kubernetes.io/projected/03049b65-310d-4ad7-b614-1daf3a50b528-kube-api-access-f62nk\") pod \"volume-data-source-validator-58c9f5f46d-nw947\" (UID: \"03049b65-310d-4ad7-b614-1daf3a50b528\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947" Oct 30 14:06:33.752019 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751919 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb9d3ab7-8a7f-4ccd-a336-912b00332110-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:33.752019 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751950 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:33.752019 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.751978 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-bound-sa-token\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.752019 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752002 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42511697-bcd4-4df3-819c-ab38ba5159c3-serving-cert\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:33.752221 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752105 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-default-certificate\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.752221 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752126 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9d3ab7-8a7f-4ccd-a336-912b00332110-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:33.752221 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752142 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4070c760-6728-4166-a333-77af48851fd8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:33.752221 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.752142 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:33.752221 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752205 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77kg8\" (UniqueName: \"kubernetes.io/projected/dfc6f25a-6a7a-472c-af78-683613ffb7a5-kube-api-access-77kg8\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.752439 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752241 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4171ff7a-5894-49d1-8683-606723fccdf9-serving-cert\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.752439 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752307 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-installation-pull-secrets\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.752439 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752359 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2zxn\" (UniqueName: \"kubernetes.io/projected/66b11a57-8746-4a93-b726-a877c2a22a98-kube-api-access-l2zxn\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.752439 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752384 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.752439 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752420 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42511697-bcd4-4df3-819c-ab38ba5159c3-config\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:33.752679 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752467 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-trusted-ca\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.752679 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752468 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.752679 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752521 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmtf\" (UniqueName: \"kubernetes.io/projected/4171ff7a-5894-49d1-8683-606723fccdf9-kube-api-access-ssmtf\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.752679 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.752531 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 14:06:33.752679 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752557 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-snapshots\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.752679 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.752579 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:34.252563531 +0000 UTC m=+31.137321795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : secret "router-metrics-certs-default" not found Oct 30 14:06:33.752679 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752563 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-stats-auth\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.753050 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752700 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-service-ca-bundle\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.753050 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752789 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-certificates\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.753050 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752817 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxk4p\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-kube-api-access-hxk4p\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.753050 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752846 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-trusted-ca\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.753050 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752902 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-928ww\" (UniqueName: \"kubernetes.io/projected/fb9d3ab7-8a7f-4ccd-a336-912b00332110-kube-api-access-928ww\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:33.753050 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752935 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.753050 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.752944 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls podName:08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:34.252929046 +0000 UTC m=+31.137687305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rt4nm" (UID: "08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65") : secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:33.753050 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.752997 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56lth\" (UniqueName: \"kubernetes.io/projected/a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5-kube-api-access-56lth\") pod \"kas-connection-checker-685895c7c9-f298s\" (UID: \"a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5\") " pod="kube-system/kas-connection-checker-685895c7c9-f298s" Oct 30 14:06:33.753050 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.753015 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:34.253003684 +0000 UTC m=+31.137761945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : configmap references non-existent config key: service-ca.crt Oct 30 14:06:33.753050 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.753045 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-tmp\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.753609 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.753120 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqjv\" (UniqueName: \"kubernetes.io/projected/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-kube-api-access-7sqjv\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:33.753609 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.753150 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrrj9\" (UniqueName: \"kubernetes.io/projected/42511697-bcd4-4df3-819c-ab38ba5159c3-kube-api-access-rrrj9\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:33.753609 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.753122 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.753609 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.753233 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-config\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.753609 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.753321 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-image-registry-private-configuration\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.753609 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.753351 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd9fc616-08fa-480b-bd77-622e4dfbf59e-ca-trust-extracted\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.753609 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.753353 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-tmp\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.754030 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.754011 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-config\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.754099 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.754083 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:33.754694 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.754668 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66b11a57-8746-4a93-b726-a877c2a22a98-serving-cert\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.754969 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.754950 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4171ff7a-5894-49d1-8683-606723fccdf9-serving-cert\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.755037 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.755017 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-default-certificate\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.755412 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.755391 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-stats-auth\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.776147 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.776117 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56lth\" (UniqueName: \"kubernetes.io/projected/a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5-kube-api-access-56lth\") pod \"kas-connection-checker-685895c7c9-f298s\" (UID: \"a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5\") " pod="kube-system/kas-connection-checker-685895c7c9-f298s" Oct 30 14:06:33.776257 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.776123 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmtf\" (UniqueName: \"kubernetes.io/projected/4171ff7a-5894-49d1-8683-606723fccdf9-kube-api-access-ssmtf\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.776374 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.776353 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2zxn\" (UniqueName: \"kubernetes.io/projected/66b11a57-8746-4a93-b726-a877c2a22a98-kube-api-access-l2zxn\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.776494 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.776476 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqjv\" (UniqueName: \"kubernetes.io/projected/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-kube-api-access-7sqjv\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:33.777203 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.777179 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77kg8\" (UniqueName: \"kubernetes.io/projected/dfc6f25a-6a7a-472c-af78-683613ffb7a5-kube-api-access-77kg8\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:33.783740 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.782685 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 14:06:33.783740 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.782694 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 14:06:33.783740 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.782760 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:33.783740 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.782764 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 14:06:33.783740 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.782802 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 14:06:33.783740 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.782856 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 14:06:33.783740 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.782926 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:33.783740 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.782983 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 14:06:33.795624 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.795605 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jmrnq" Oct 30 14:06:33.808803 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.808783 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dmj28" Oct 30 14:06:33.817240 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.817213 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 14:06:33.817240 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.817222 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 14:06:33.817439 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.817403 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7x2qs\"" Oct 30 14:06:33.817481 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.817444 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4w84p\"" Oct 30 14:06:33.845410 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.845390 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f298s" Oct 30 14:06:33.853680 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.853657 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.853790 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.853697 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-image-registry-private-configuration\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.853790 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.853739 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd9fc616-08fa-480b-bd77-622e4dfbf59e-ca-trust-extracted\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.853887 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.853791 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/911b39fc-f2e6-4094-9be0-68606d2aa4f1-cert\") pod \"ingress-canary-m8jt7\" (UID: \"911b39fc-f2e6-4094-9be0-68606d2aa4f1\") " pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:33.853887 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.853825 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:33.853979 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.853927 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:33.853979 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.853952 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thjb5\" (UniqueName: \"kubernetes.io/projected/4070c760-6728-4166-a333-77af48851fd8-kube-api-access-thjb5\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:33.854085 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.853984 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-image-registry-private-configuration\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.854284 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854199 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd9fc616-08fa-480b-bd77-622e4dfbf59e-ca-trust-extracted\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.854284 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854205 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62nk\" (UniqueName: \"kubernetes.io/projected/03049b65-310d-4ad7-b614-1daf3a50b528-kube-api-access-f62nk\") pod \"volume-data-source-validator-58c9f5f46d-nw947\" (UID: \"03049b65-310d-4ad7-b614-1daf3a50b528\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947" Oct 30 14:06:33.854284 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854234 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb9d3ab7-8a7f-4ccd-a336-912b00332110-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:33.854420 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.854288 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 14:06:33.854420 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.854359 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls podName:4070c760-6728-4166-a333-77af48851fd8 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:34.354339246 +0000 UTC m=+31.239097505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zrb4p" (UID: "4070c760-6728-4166-a333-77af48851fd8") : secret "samples-operator-tls" not found Oct 30 14:06:33.854525 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854439 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv27n\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-kube-api-access-jv27n\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.854525 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854492 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-bound-sa-token\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.854636 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854615 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42511697-bcd4-4df3-819c-ab38ba5159c3-serving-cert\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:33.854688 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854661 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bmb\" (UniqueName: \"kubernetes.io/projected/17aed83f-3fbe-4b2e-8714-e61122539496-kube-api-access-82bmb\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.854760 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854694 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9d3ab7-8a7f-4ccd-a336-912b00332110-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:33.854760 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854737 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4070c760-6728-4166-a333-77af48851fd8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:33.854869 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854776 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-installation-pull-secrets\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.854869 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854810 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-trusted-ca\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.854869 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854838 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.854869 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854863 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17aed83f-3fbe-4b2e-8714-e61122539496-config-volume\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.855050 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854866 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb9d3ab7-8a7f-4ccd-a336-912b00332110-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:33.855050 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854892 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lztlh\" (UniqueName: \"kubernetes.io/projected/fe78a7b6-cba1-4c52-b190-0febeaab5035-kube-api-access-lztlh\") pod \"network-check-source-6f78d59f99-2sxp4\" (UID: \"fe78a7b6-cba1-4c52-b190-0febeaab5035\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4" Oct 30 14:06:33.855050 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.854922 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42511697-bcd4-4df3-819c-ab38ba5159c3-config\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:33.855422 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855390 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd07a434-a5f6-4cda-a93d-8824978aa590-ca-trust-extracted\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.855509 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855449 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-certificates\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.855509 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855479 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxk4p\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-kube-api-access-hxk4p\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.855610 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855544 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-installation-pull-secrets\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.855610 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855568 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4070c760-6728-4166-a333-77af48851fd8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:33.855610 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855579 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-trusted-ca\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.855787 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855616 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-928ww\" (UniqueName: \"kubernetes.io/projected/fb9d3ab7-8a7f-4ccd-a336-912b00332110-kube-api-access-928ww\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:33.855787 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855617 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42511697-bcd4-4df3-819c-ab38ba5159c3-config\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:33.855787 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855645 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vklh\" (UniqueName: \"kubernetes.io/projected/911b39fc-f2e6-4094-9be0-68606d2aa4f1-kube-api-access-5vklh\") pod \"ingress-canary-m8jt7\" (UID: \"911b39fc-f2e6-4094-9be0-68606d2aa4f1\") " pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:33.855946 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.855814 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:33.855946 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.855830 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fff9cc46-btf9d: secret "image-registry-tls" not found Oct 30 14:06:33.855946 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855832 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17aed83f-3fbe-4b2e-8714-e61122539496-tmp-dir\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.855946 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.855892 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls podName:bd9fc616-08fa-480b-bd77-622e4dfbf59e nodeName:}" failed. No retries permitted until 2026-10-30 14:06:34.355877806 +0000 UTC m=+31.240636057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls") pod "image-registry-8fff9cc46-btf9d" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e") : secret "image-registry-tls" not found Oct 30 14:06:33.855946 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855935 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-certificates\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.856187 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.855974 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-bound-sa-token\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.856187 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.856012 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.856187 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.856058 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2536dd8-0f2c-4571-b0db-484894b2a99f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:33.856187 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.856123 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrj9\" (UniqueName: \"kubernetes.io/projected/42511697-bcd4-4df3-819c-ab38ba5159c3-kube-api-access-rrrj9\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:33.856187 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.856127 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-certificates\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.856979 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.856934 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-trusted-ca\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.857232 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.857215 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42511697-bcd4-4df3-819c-ab38ba5159c3-serving-cert\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:33.858249 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.858223 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-image-registry-private-configuration\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.858375 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.858358 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-installation-pull-secrets\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.858506 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.858488 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9d3ab7-8a7f-4ccd-a336-912b00332110-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:33.867619 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.867567 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:33.879426 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.879394 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrj9\" (UniqueName: \"kubernetes.io/projected/42511697-bcd4-4df3-819c-ab38ba5159c3-kube-api-access-rrrj9\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:33.880018 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.879788 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxk4p\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-kube-api-access-hxk4p\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.880018 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.879806 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-928ww\" (UniqueName: \"kubernetes.io/projected/fb9d3ab7-8a7f-4ccd-a336-912b00332110-kube-api-access-928ww\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:33.880018 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.879985 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-bound-sa-token\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:33.882107 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.882065 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thjb5\" (UniqueName: \"kubernetes.io/projected/4070c760-6728-4166-a333-77af48851fd8-kube-api-access-thjb5\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:33.883124 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.883106 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62nk\" (UniqueName: \"kubernetes.io/projected/03049b65-310d-4ad7-b614-1daf3a50b528-kube-api-access-f62nk\") pod \"volume-data-source-validator-58c9f5f46d-nw947\" (UID: \"03049b65-310d-4ad7-b614-1daf3a50b528\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947" Oct 30 14:06:33.889560 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.889539 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:33.938283 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.938034 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:33.956535 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.953940 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.956622 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/911b39fc-f2e6-4094-9be0-68606d2aa4f1-cert\") pod \"ingress-canary-m8jt7\" (UID: \"911b39fc-f2e6-4094-9be0-68606d2aa4f1\") " pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.956691 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.956759 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-image-registry-private-configuration\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.956800 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv27n\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-kube-api-access-jv27n\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.956841 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bmb\" (UniqueName: \"kubernetes.io/projected/17aed83f-3fbe-4b2e-8714-e61122539496-kube-api-access-82bmb\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.956876 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-trusted-ca\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.956910 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17aed83f-3fbe-4b2e-8714-e61122539496-config-volume\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.956937 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztlh\" (UniqueName: \"kubernetes.io/projected/fe78a7b6-cba1-4c52-b190-0febeaab5035-kube-api-access-lztlh\") pod \"network-check-source-6f78d59f99-2sxp4\" (UID: \"fe78a7b6-cba1-4c52-b190-0febeaab5035\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.956995 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd07a434-a5f6-4cda-a93d-8824978aa590-ca-trust-extracted\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.957038 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-installation-pull-secrets\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.957069 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vklh\" (UniqueName: \"kubernetes.io/projected/911b39fc-f2e6-4094-9be0-68606d2aa4f1-kube-api-access-5vklh\") pod \"ingress-canary-m8jt7\" (UID: \"911b39fc-f2e6-4094-9be0-68606d2aa4f1\") " pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.957102 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17aed83f-3fbe-4b2e-8714-e61122539496-tmp-dir\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.957144 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-certificates\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.957175 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-bound-sa-token\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.957198 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.958854 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.957224 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2536dd8-0f2c-4571-b0db-484894b2a99f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.957269 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.957410 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.957424 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89594b78c-qxvnh: secret "image-registry-tls" not found Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.957479 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls podName:fd07a434-a5f6-4cda-a93d-8824978aa590 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:34.457460463 +0000 UTC m=+31.342218727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls") pod "image-registry-89594b78c-qxvnh" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590") : secret "image-registry-tls" not found Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.958085 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17aed83f-3fbe-4b2e-8714-e61122539496-tmp-dir\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.958266 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17aed83f-3fbe-4b2e-8714-e61122539496-config-volume\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.958347 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-trusted-ca\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.958455 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.958507 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert podName:c2536dd8-0f2c-4571-b0db-484894b2a99f nodeName:}" failed. No retries permitted until 2026-10-30 14:06:34.458490101 +0000 UTC m=+31.343248359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9s2h" (UID: "c2536dd8-0f2c-4571-b0db-484894b2a99f") : secret "networking-console-plugin-cert" not found Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.958636 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd07a434-a5f6-4cda-a93d-8824978aa590-ca-trust-extracted\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.959032 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.959227 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-certificates\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.959895 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2536dd8-0f2c-4571-b0db-484894b2a99f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:33.960148 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:33.960008 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls podName:17aed83f-3fbe-4b2e-8714-e61122539496 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:34.459989966 +0000 UTC m=+31.344748231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls") pod "dns-default-fvkvc" (UID: "17aed83f-3fbe-4b2e-8714-e61122539496") : secret "dns-default-metrics-tls" not found Oct 30 14:06:33.962525 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.962337 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-image-registry-private-configuration\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.970070 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.970021 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/911b39fc-f2e6-4094-9be0-68606d2aa4f1-cert\") pod \"ingress-canary-m8jt7\" (UID: \"911b39fc-f2e6-4094-9be0-68606d2aa4f1\") " pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:33.971030 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.970992 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-installation-pull-secrets\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.986153 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.986066 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bmb\" (UniqueName: \"kubernetes.io/projected/17aed83f-3fbe-4b2e-8714-e61122539496-kube-api-access-82bmb\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:33.988256 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.987920 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vklh\" (UniqueName: \"kubernetes.io/projected/911b39fc-f2e6-4094-9be0-68606d2aa4f1-kube-api-access-5vklh\") pod \"ingress-canary-m8jt7\" (UID: \"911b39fc-f2e6-4094-9be0-68606d2aa4f1\") " pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:33.989098 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.989067 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-bound-sa-token\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:33.993757 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.992893 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztlh\" (UniqueName: \"kubernetes.io/projected/fe78a7b6-cba1-4c52-b190-0febeaab5035-kube-api-access-lztlh\") pod \"network-check-source-6f78d59f99-2sxp4\" (UID: \"fe78a7b6-cba1-4c52-b190-0febeaab5035\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4" Oct 30 14:06:33.995548 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:33.995301 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv27n\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-kube-api-access-jv27n\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:34.039738 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.038779 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947" Oct 30 14:06:34.074791 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.074761 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4" Oct 30 14:06:34.087958 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.087932 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:34.175344 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.175321 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jmrnq"] Oct 30 14:06:34.178254 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.178165 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dmj28"] Oct 30 14:06:34.184907 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.184844 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f298s"] Oct 30 14:06:34.239521 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.239256 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-84mcb"] Oct 30 14:06:34.247770 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.242512 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tfvs9"] Oct 30 14:06:34.255030 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:34.254915 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b11a57_8746_4a93_b726_a877c2a22a98.slice/crio-a9ce8bbfd5ed5fcbbf19934621bfc94b2d35d455fb7ebf77f4b4c86bb21ae612 WatchSource:0}: Error finding container a9ce8bbfd5ed5fcbbf19934621bfc94b2d35d455fb7ebf77f4b4c86bb21ae612: Status 404 returned error can't find the container with id a9ce8bbfd5ed5fcbbf19934621bfc94b2d35d455fb7ebf77f4b4c86bb21ae612 Oct 30 14:06:34.266962 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.265591 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:34.266962 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.265650 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:34.266962 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.265790 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:34.266962 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.265946 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:34.266962 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.266000 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls podName:08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:35.265982604 +0000 UTC m=+32.150740864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rt4nm" (UID: "08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65") : secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:34.266962 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.266062 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 14:06:34.266962 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.266096 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:35.266085348 +0000 UTC m=+32.150843606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : secret "router-metrics-certs-default" not found Oct 30 14:06:34.266962 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.266165 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:35.266153072 +0000 UTC m=+32.150911330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : configmap references non-existent config key: service-ca.crt Oct 30 14:06:34.274766 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.274582 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh"] Oct 30 14:06:34.295652 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:34.295594 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9d3ab7_8a7f_4ccd_a336_912b00332110.slice/crio-ed8f58363bcee5fbb7ff703de95bf9882033f5112ca4510aa71c383aa4acbd6f WatchSource:0}: Error finding container ed8f58363bcee5fbb7ff703de95bf9882033f5112ca4510aa71c383aa4acbd6f: Status 404 returned error can't find the container with id ed8f58363bcee5fbb7ff703de95bf9882033f5112ca4510aa71c383aa4acbd6f Oct 30 14:06:34.301006 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.300706 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz"] Oct 30 14:06:34.309511 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:34.309484 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42511697_bcd4_4df3_819c_ab38ba5159c3.slice/crio-b5e718a5497402d58bac6780c3a2e6c291eede489a5938ded3087c0e77797ddd WatchSource:0}: Error finding container b5e718a5497402d58bac6780c3a2e6c291eede489a5938ded3087c0e77797ddd: Status 404 returned error can't find the container with id b5e718a5497402d58bac6780c3a2e6c291eede489a5938ded3087c0e77797ddd Oct 30 14:06:34.349146 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.349119 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947"] Oct 30 14:06:34.353032 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:34.353009 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03049b65_310d_4ad7_b614_1daf3a50b528.slice/crio-4079ca19c89bdeaaf49de19f510759b96f2c4ba34a2b40ea6fcbe96b86afc2fd WatchSource:0}: Error finding container 4079ca19c89bdeaaf49de19f510759b96f2c4ba34a2b40ea6fcbe96b86afc2fd: Status 404 returned error can't find the container with id 4079ca19c89bdeaaf49de19f510759b96f2c4ba34a2b40ea6fcbe96b86afc2fd Oct 30 14:06:34.366428 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.366243 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:34.366428 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.366346 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:34.366428 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.366390 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 14:06:34.366626 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.366441 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls podName:4070c760-6728-4166-a333-77af48851fd8 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:35.366427341 +0000 UTC m=+32.251185590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zrb4p" (UID: "4070c760-6728-4166-a333-77af48851fd8") : secret "samples-operator-tls" not found Oct 30 14:06:34.366626 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.366499 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:34.366626 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.366518 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fff9cc46-btf9d: secret "image-registry-tls" not found Oct 30 14:06:34.366626 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.366566 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls podName:bd9fc616-08fa-480b-bd77-622e4dfbf59e nodeName:}" failed. No retries permitted until 2026-10-30 14:06:35.366551326 +0000 UTC m=+32.251309585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls") pod "image-registry-8fff9cc46-btf9d" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e") : secret "image-registry-tls" not found Oct 30 14:06:34.395692 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.395640 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4"] Oct 30 14:06:34.399837 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:34.399810 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe78a7b6_cba1_4c52_b190_0febeaab5035.slice/crio-9e3af484b38bfa7e08e90b016ba031ea62141d0b5e580f3e77d688b42155a0d1 WatchSource:0}: Error finding container 9e3af484b38bfa7e08e90b016ba031ea62141d0b5e580f3e77d688b42155a0d1: Status 404 returned error can't find the container with id 9e3af484b38bfa7e08e90b016ba031ea62141d0b5e580f3e77d688b42155a0d1 Oct 30 14:06:34.421334 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.421312 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m8jt7"] Oct 30 14:06:34.424764 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:34.424741 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911b39fc_f2e6_4094_9be0_68606d2aa4f1.slice/crio-5e7a18c0a9391981fba99e22e260ba8b224ed58dc92119c0753c231079b552ca WatchSource:0}: Error finding container 5e7a18c0a9391981fba99e22e260ba8b224ed58dc92119c0753c231079b552ca: Status 404 returned error can't find the container with id 5e7a18c0a9391981fba99e22e260ba8b224ed58dc92119c0753c231079b552ca Oct 30 14:06:34.467691 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.467662 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:34.467820 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.467709 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:34.467820 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.467779 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:34.467820 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.467810 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 14:06:34.467972 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.467856 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls podName:17aed83f-3fbe-4b2e-8714-e61122539496 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:35.467842674 +0000 UTC m=+32.352600947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls") pod "dns-default-fvkvc" (UID: "17aed83f-3fbe-4b2e-8714-e61122539496") : secret "dns-default-metrics-tls" not found Oct 30 14:06:34.467972 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.467880 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:34.467972 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.467898 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89594b78c-qxvnh: secret "image-registry-tls" not found Oct 30 14:06:34.467972 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.467913 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 14:06:34.467972 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.467938 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls podName:fd07a434-a5f6-4cda-a93d-8824978aa590 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:35.467926929 +0000 UTC m=+32.352685185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls") pod "image-registry-89594b78c-qxvnh" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590") : secret "image-registry-tls" not found Oct 30 14:06:34.467972 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:34.467959 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert podName:c2536dd8-0f2c-4571-b0db-484894b2a99f nodeName:}" failed. No retries permitted until 2026-10-30 14:06:35.467946106 +0000 UTC m=+32.352704364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9s2h" (UID: "c2536dd8-0f2c-4571-b0db-484894b2a99f") : secret "networking-console-plugin-cert" not found Oct 30 14:06:34.776375 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.776294 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:34.776375 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.776352 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:34.776573 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.776302 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:34.791490 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.791463 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 14:06:34.791883 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.791735 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z4lqh\"" Oct 30 14:06:34.791883 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.791777 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 14:06:34.791883 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.791823 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2dqn6\"" Oct 30 14:06:34.792031 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:34.791904 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgz8z\"" Oct 30 14:06:35.024194 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.023108 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" event={"ID":"fb9d3ab7-8a7f-4ccd-a336-912b00332110","Type":"ContainerStarted","Data":"ed8f58363bcee5fbb7ff703de95bf9882033f5112ca4510aa71c383aa4acbd6f"} Oct 30 14:06:35.028847 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.028741 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" event={"ID":"66b11a57-8746-4a93-b726-a877c2a22a98","Type":"ContainerStarted","Data":"a9ce8bbfd5ed5fcbbf19934621bfc94b2d35d455fb7ebf77f4b4c86bb21ae612"} Oct 30 14:06:35.031843 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.031101 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f298s" event={"ID":"a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5","Type":"ContainerStarted","Data":"ccfcf78740b132e30d0bd817dffc1d44c6f446e7afeb3c34e568f1ea4f6c23bc"} Oct 30 14:06:35.031843 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.031143 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f298s" event={"ID":"a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5","Type":"ContainerStarted","Data":"52aef7cad4bd812500483db121cfae723767e4715bcea840fb8cdb3305c03e9c"} Oct 30 14:06:35.034055 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.033557 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jmrnq" event={"ID":"1cdffa3f-7378-4f50-b1bc-e87aa647c4d8","Type":"ContainerStarted","Data":"7bb004b60e805c35852b2fc72639b095b9f87fa6dc7b05c5c43736b89d0aec18"} Oct 30 14:06:35.034055 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.033596 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jmrnq" event={"ID":"1cdffa3f-7378-4f50-b1bc-e87aa647c4d8","Type":"ContainerStarted","Data":"b612015fb112ba2dbc4456abd28335ab8c9f6ec986d7e3f0991c77b02c7502eb"} Oct 30 14:06:35.036818 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.036704 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m8jt7" event={"ID":"911b39fc-f2e6-4094-9be0-68606d2aa4f1","Type":"ContainerStarted","Data":"5e7a18c0a9391981fba99e22e260ba8b224ed58dc92119c0753c231079b552ca"} Oct 30 14:06:35.040413 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.040353 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" event={"ID":"42511697-bcd4-4df3-819c-ab38ba5159c3","Type":"ContainerStarted","Data":"b5e718a5497402d58bac6780c3a2e6c291eede489a5938ded3087c0e77797ddd"} Oct 30 14:06:35.042247 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.042224 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tfvs9" event={"ID":"4171ff7a-5894-49d1-8683-606723fccdf9","Type":"ContainerStarted","Data":"51b981c69d8492be055c2dc1d8f7be2b52ce30979a988059582fc6a79a73b714"} Oct 30 14:06:35.047581 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.047553 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dmj28" event={"ID":"3829121e-3790-47ff-b827-8774a70fec04","Type":"ContainerStarted","Data":"4707007256f65e980325b8c74293440a4036a3d8051aa1d9c519c74fa379e1e4"} Oct 30 14:06:35.047676 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.047603 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dmj28" event={"ID":"3829121e-3790-47ff-b827-8774a70fec04","Type":"ContainerStarted","Data":"bd6bd9f6c7a2ec2513e595e95e160b238a68d219ae90dcf25d7b31dc58fe390e"} Oct 30 14:06:35.053552 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.053485 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4" event={"ID":"fe78a7b6-cba1-4c52-b190-0febeaab5035","Type":"ContainerStarted","Data":"9e3af484b38bfa7e08e90b016ba031ea62141d0b5e580f3e77d688b42155a0d1"} Oct 30 14:06:35.056244 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.056195 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947" event={"ID":"03049b65-310d-4ad7-b614-1daf3a50b528","Type":"ContainerStarted","Data":"4079ca19c89bdeaaf49de19f510759b96f2c4ba34a2b40ea6fcbe96b86afc2fd"} Oct 30 14:06:35.073089 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.073038 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-f298s" podStartSLOduration=326.073020856 podStartE2EDuration="5m26.073020856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:35.071357606 +0000 UTC m=+31.956115875" watchObservedRunningTime="2026-10-30 14:06:35.073020856 +0000 UTC m=+31.957779124" Oct 30 14:06:35.154378 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.154300 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dmj28" podStartSLOduration=326.154281943 podStartE2EDuration="5m26.154281943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:35.112235438 +0000 UTC m=+31.996993704" watchObservedRunningTime="2026-10-30 14:06:35.154281943 +0000 UTC m=+32.039040223" Oct 30 14:06:35.154652 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.154611 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jmrnq" podStartSLOduration=326.154603148 podStartE2EDuration="5m26.154603148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:35.15253362 +0000 UTC m=+32.037291880" watchObservedRunningTime="2026-10-30 14:06:35.154603148 +0000 UTC m=+32.039361411" Oct 30 14:06:35.276233 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.276194 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:35.276423 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.276395 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:35.276538 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.276464 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls podName:08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:37.276445873 +0000 UTC m=+34.161204134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rt4nm" (UID: "08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65") : secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:35.276538 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.276478 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 14:06:35.276648 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.276546 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:37.276527925 +0000 UTC m=+34.161286181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : secret "router-metrics-certs-default" not found Oct 30 14:06:35.276947 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.276918 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:35.277058 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.276998 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:35.277188 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.277169 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:37.277155191 +0000 UTC m=+34.161913450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : configmap references non-existent config key: service-ca.crt Oct 30 14:06:35.379259 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.378395 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:35.379259 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.378575 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:35.379259 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.378751 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 14:06:35.379259 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.378808 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls podName:4070c760-6728-4166-a333-77af48851fd8 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:37.378795379 +0000 UTC m=+34.263553635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zrb4p" (UID: "4070c760-6728-4166-a333-77af48851fd8") : secret "samples-operator-tls" not found Oct 30 14:06:35.379259 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.379165 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:35.379259 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.379178 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fff9cc46-btf9d: secret "image-registry-tls" not found Oct 30 14:06:35.379259 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.379220 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls podName:bd9fc616-08fa-480b-bd77-622e4dfbf59e nodeName:}" failed. No retries permitted until 2026-10-30 14:06:37.379206834 +0000 UTC m=+34.263965083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls") pod "image-registry-8fff9cc46-btf9d" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e") : secret "image-registry-tls" not found Oct 30 14:06:35.480756 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.479925 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:35.480756 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.479992 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:35.480756 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:35.480034 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:35.480756 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.480186 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 14:06:35.480756 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.480246 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert podName:c2536dd8-0f2c-4571-b0db-484894b2a99f nodeName:}" failed. No retries permitted until 2026-10-30 14:06:37.480230023 +0000 UTC m=+34.364988269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9s2h" (UID: "c2536dd8-0f2c-4571-b0db-484894b2a99f") : secret "networking-console-plugin-cert" not found Oct 30 14:06:35.480756 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.480588 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 14:06:35.480756 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.480625 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls podName:17aed83f-3fbe-4b2e-8714-e61122539496 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:37.48061483 +0000 UTC m=+34.365373073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls") pod "dns-default-fvkvc" (UID: "17aed83f-3fbe-4b2e-8714-e61122539496") : secret "dns-default-metrics-tls" not found Oct 30 14:06:35.480756 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.480684 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:35.480756 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.480693 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89594b78c-qxvnh: secret "image-registry-tls" not found Oct 30 14:06:35.480756 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:35.480727 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls podName:fd07a434-a5f6-4cda-a93d-8824978aa590 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:37.480708718 +0000 UTC m=+34.365466962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls") pod "image-registry-89594b78c-qxvnh" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590") : secret "image-registry-tls" not found Oct 30 14:06:37.068879 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.068843 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947" event={"ID":"03049b65-310d-4ad7-b614-1daf3a50b528","Type":"ContainerStarted","Data":"c664964333ffd7771b798b8ec939fe2c816c1d8b4138ab3129ed39497aea3e35"} Oct 30 14:06:37.301211 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.301121 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:37.301362 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.301214 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:37.301362 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.301266 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:37.301467 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.301440 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:41.301420258 +0000 UTC m=+38.186178521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : configmap references non-existent config key: service-ca.crt Oct 30 14:06:37.301906 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.301883 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:37.302028 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.301947 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls podName:08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:41.301932698 +0000 UTC m=+38.186690941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rt4nm" (UID: "08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65") : secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:37.302028 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.302013 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 14:06:37.302146 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.302047 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:41.302036836 +0000 UTC m=+38.186795088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : secret "router-metrics-certs-default" not found Oct 30 14:06:37.402351 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.402317 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:37.402530 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.402373 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:37.402530 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.402401 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:37.402530 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.402516 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:37.402694 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.402653 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 14:06:37.402772 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.402698 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls podName:4070c760-6728-4166-a333-77af48851fd8 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:41.402686456 +0000 UTC m=+38.287444713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zrb4p" (UID: "4070c760-6728-4166-a333-77af48851fd8") : secret "samples-operator-tls" not found Oct 30 14:06:37.403034 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.403015 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:37.403034 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.403034 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fff9cc46-btf9d: secret "image-registry-tls" not found Oct 30 14:06:37.403182 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.403068 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls podName:bd9fc616-08fa-480b-bd77-622e4dfbf59e nodeName:}" failed. No retries permitted until 2026-10-30 14:06:41.403058542 +0000 UTC m=+38.287816799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls") pod "image-registry-8fff9cc46-btf9d" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e") : secret "image-registry-tls" not found Oct 30 14:06:37.403465 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.403448 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 14:06:37.403549 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.403514 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs podName:a14b5c29-8cac-42f8-a3b2-6bf636cc05a4 nodeName:}" failed. No retries permitted until 2026-10-30 14:07:09.403497299 +0000 UTC m=+66.288255559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs") pod "network-metrics-daemon-rgjxr" (UID: "a14b5c29-8cac-42f8-a3b2-6bf636cc05a4") : secret "metrics-daemon-secret" not found Oct 30 14:06:37.414665 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.414638 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:37.503053 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.503007 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vhn\" (UniqueName: \"kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn\") pod \"network-check-target-stdbh\" (UID: \"aea273bc-03a8-41ad-8ed0-c1eab9a23a45\") " pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:37.503228 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.503065 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:37.503228 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.503107 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:37.503228 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.503152 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:37.503413 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.503393 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 14:06:37.503564 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.503462 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert podName:c2536dd8-0f2c-4571-b0db-484894b2a99f nodeName:}" failed. No retries permitted until 2026-10-30 14:06:41.503442538 +0000 UTC m=+38.388200789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9s2h" (UID: "c2536dd8-0f2c-4571-b0db-484894b2a99f") : secret "networking-console-plugin-cert" not found Oct 30 14:06:37.504393 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.504292 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:37.504393 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.504313 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89594b78c-qxvnh: secret "image-registry-tls" not found Oct 30 14:06:37.504393 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.504339 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 14:06:37.504393 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.504368 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls podName:fd07a434-a5f6-4cda-a93d-8824978aa590 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:41.504352884 +0000 UTC m=+38.389111132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls") pod "image-registry-89594b78c-qxvnh" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590") : secret "image-registry-tls" not found Oct 30 14:06:37.504668 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:37.504404 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls podName:17aed83f-3fbe-4b2e-8714-e61122539496 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:41.504386479 +0000 UTC m=+38.389144743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls") pod "dns-default-fvkvc" (UID: "17aed83f-3fbe-4b2e-8714-e61122539496") : secret "dns-default-metrics-tls" not found Oct 30 14:06:37.513363 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.513338 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vhn\" (UniqueName: \"kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn\") pod \"network-check-target-stdbh\" (UID: \"aea273bc-03a8-41ad-8ed0-c1eab9a23a45\") " pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:37.521116 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.520937 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:37.808098 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.807875 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:37.855748 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.854906 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947" podStartSLOduration=300.483527268 podStartE2EDuration="5m2.854884617s" totalImagesPullingTime="2.371357349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:37.110319425 +0000 UTC m=+33.995077690" watchObservedRunningTime="2026-10-30 14:06:37.854884617 +0000 UTC m=+34.739642900" Oct 30 14:06:37.855748 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:37.855641 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vlmdd"] Oct 30 14:06:38.075835 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:38.075701 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vlmdd" event={"ID":"f6ae2497-ab63-4964-940a-e870d9131ae3","Type":"ContainerStarted","Data":"4cc6f99f8e3efe6cf211c29d73d271b4b53c9b3d5936ef0937345119ff3b05d6"} Oct 30 14:06:38.177988 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:38.177951 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-stdbh"] Oct 30 14:06:38.185738 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:38.185681 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea273bc_03a8_41ad_8ed0_c1eab9a23a45.slice/crio-c2b7943ae35026336881dcd417bae55e6e08df8ebcd6773537471075a3fa92d9 WatchSource:0}: Error finding container c2b7943ae35026336881dcd417bae55e6e08df8ebcd6773537471075a3fa92d9: Status 404 returned error can't find the container with id c2b7943ae35026336881dcd417bae55e6e08df8ebcd6773537471075a3fa92d9 Oct 30 14:06:39.081228 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.080988 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/0.log" Oct 30 14:06:39.081228 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.081040 2584 generic.go:356] "Generic (PLEG): container finished" podID="66b11a57-8746-4a93-b726-a877c2a22a98" containerID="3e908616fe74060f9874afe2301ba212a6c63c3cd462fa3eed80de51e5175e34" exitCode=255 Oct 30 14:06:39.081228 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.081141 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" event={"ID":"66b11a57-8746-4a93-b726-a877c2a22a98","Type":"ContainerDied","Data":"3e908616fe74060f9874afe2301ba212a6c63c3cd462fa3eed80de51e5175e34"} Oct 30 14:06:39.081749 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.081360 2584 scope.go:122] "RemoveContainer" containerID="3e908616fe74060f9874afe2301ba212a6c63c3cd462fa3eed80de51e5175e34" Oct 30 14:06:39.083503 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.083474 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-stdbh" event={"ID":"aea273bc-03a8-41ad-8ed0-c1eab9a23a45","Type":"ContainerStarted","Data":"c2b7943ae35026336881dcd417bae55e6e08df8ebcd6773537471075a3fa92d9"} Oct 30 14:06:39.085974 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.085539 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tfvs9" event={"ID":"4171ff7a-5894-49d1-8683-606723fccdf9","Type":"ContainerStarted","Data":"8b38598b9ca7000950dca8428e815e2adbf7d5911fb9245af8e1ea092224ceb1"} Oct 30 14:06:39.166251 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.166193 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-tfvs9" podStartSLOduration=299.695684596 podStartE2EDuration="5m4.166175396s" totalImagesPullingTime="4.4704908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:39.165273627 +0000 UTC m=+36.050031910" watchObservedRunningTime="2026-10-30 14:06:39.166175396 +0000 UTC m=+36.050933665" Oct 30 14:06:39.771172 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.771135 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hmtlt"] Oct 30 14:06:39.783371 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.783313 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.803319 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.803249 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hmtlt"] Oct 30 14:06:39.813586 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.813240 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 14:06:39.813586 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.813535 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 14:06:39.813898 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.813658 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g7grg\"" Oct 30 14:06:39.841755 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.839818 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-crio-socket\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.841755 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.839909 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.841755 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.839965 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.841755 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.840016 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-data-volume\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.841755 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.840070 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqwzb\" (UniqueName: \"kubernetes.io/projected/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-api-access-gqwzb\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.940815 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.940785 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-data-volume\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.940931 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.940875 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqwzb\" (UniqueName: \"kubernetes.io/projected/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-api-access-gqwzb\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.940969 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.940951 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-crio-socket\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.941026 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.941014 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.941073 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.941063 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.941166 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:39.941154 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 14:06:39.941220 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:39.941211 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:40.441192713 +0000 UTC m=+37.325950970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : configmap "kube-rbac-proxy" not found Oct 30 14:06:39.941661 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.941644 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-data-volume\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.942156 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.942106 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-crio-socket\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:39.942156 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:39.942139 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 14:06:39.942319 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:39.942195 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:40.44218028 +0000 UTC m=+37.326938530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : secret "insights-runtime-extractor-tls" not found Oct 30 14:06:39.964691 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:39.964668 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqwzb\" (UniqueName: \"kubernetes.io/projected/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-api-access-gqwzb\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:40.123022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.114869 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" event={"ID":"fb9d3ab7-8a7f-4ccd-a336-912b00332110","Type":"ContainerStarted","Data":"fa73e14891af6c9f8196a20bcfd38911d9a5116d5269f6a7a47df2b46184dce2"} Oct 30 14:06:40.123022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.116902 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/1.log" Oct 30 14:06:40.123022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.117502 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/0.log" Oct 30 14:06:40.123022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.117541 2584 generic.go:356] "Generic (PLEG): container finished" podID="66b11a57-8746-4a93-b726-a877c2a22a98" containerID="6f398efb0490e4b545d0c2f056c5615256be15aa38b06eb15dad2a77ff08d592" exitCode=255 Oct 30 14:06:40.123022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.117660 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" event={"ID":"66b11a57-8746-4a93-b726-a877c2a22a98","Type":"ContainerDied","Data":"6f398efb0490e4b545d0c2f056c5615256be15aa38b06eb15dad2a77ff08d592"} Oct 30 14:06:40.123022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.117728 2584 scope.go:122] "RemoveContainer" containerID="3e908616fe74060f9874afe2301ba212a6c63c3cd462fa3eed80de51e5175e34" Oct 30 14:06:40.123022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.118014 2584 scope.go:122] "RemoveContainer" containerID="6f398efb0490e4b545d0c2f056c5615256be15aa38b06eb15dad2a77ff08d592" Oct 30 14:06:40.123022 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:40.118277 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-84mcb_openshift-console-operator(66b11a57-8746-4a93-b726-a877c2a22a98)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" podUID="66b11a57-8746-4a93-b726-a877c2a22a98" Oct 30 14:06:40.123022 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.122469 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m8jt7" event={"ID":"911b39fc-f2e6-4094-9be0-68606d2aa4f1","Type":"ContainerStarted","Data":"ebf89feb26074ea95ed4d2f4efd69dda7b50e42e97cad48ea5d372b10987343f"} Oct 30 14:06:40.125746 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.124777 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" event={"ID":"42511697-bcd4-4df3-819c-ab38ba5159c3","Type":"ContainerStarted","Data":"3d6462a55e3ea8c88897e824672873916f2dc2e4a50c2e9358c7e98922c92f8c"} Oct 30 14:06:40.157591 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.157548 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" podStartSLOduration=299.88517667 podStartE2EDuration="5m5.157536543s" totalImagesPullingTime="5.272359873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:40.157026795 +0000 UTC m=+37.041785061" watchObservedRunningTime="2026-10-30 14:06:40.157536543 +0000 UTC m=+37.042294809" Oct 30 14:06:40.294952 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.293187 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m8jt7" podStartSLOduration=2.13532965 podStartE2EDuration="7.293166759s" totalImagesPullingTime="5.157837109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:40.292399144 +0000 UTC m=+37.177157412" watchObservedRunningTime="2026-10-30 14:06:40.293166759 +0000 UTC m=+37.177925025" Oct 30 14:06:40.294952 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.293667 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" podStartSLOduration=300.033599247 podStartE2EDuration="5m5.293657068s" totalImagesPullingTime="5.260057821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:40.244855528 +0000 UTC m=+37.129613795" watchObservedRunningTime="2026-10-30 14:06:40.293657068 +0000 UTC m=+37.178415338" Oct 30 14:06:40.447741 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.447060 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:40.447741 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:40.447136 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:40.447741 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:40.447235 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 14:06:40.447741 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:40.447306 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:41.447289105 +0000 UTC m=+38.332047364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : configmap "kube-rbac-proxy" not found Oct 30 14:06:40.447741 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:40.447653 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 14:06:40.447741 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:40.447693 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:41.447682009 +0000 UTC m=+38.332440266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : secret "insights-runtime-extractor-tls" not found Oct 30 14:06:41.131678 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.130989 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4" event={"ID":"fe78a7b6-cba1-4c52-b190-0febeaab5035","Type":"ContainerStarted","Data":"d47ec2d622f4fe6a828684a61fb8e61ebec7ff183bbbe95d100e3b5058f81f9d"} Oct 30 14:06:41.135977 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.133521 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/1.log" Oct 30 14:06:41.135977 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.134087 2584 scope.go:122] "RemoveContainer" containerID="6f398efb0490e4b545d0c2f056c5615256be15aa38b06eb15dad2a77ff08d592" Oct 30 14:06:41.135977 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.134301 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-84mcb_openshift-console-operator(66b11a57-8746-4a93-b726-a877c2a22a98)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" podUID="66b11a57-8746-4a93-b726-a877c2a22a98" Oct 30 14:06:41.146125 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.145102 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv"] Oct 30 14:06:41.172502 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.172445 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4" podStartSLOduration=296.589586666 podStartE2EDuration="5m2.172428333s" totalImagesPullingTime="5.582841667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:41.171536152 +0000 UTC m=+38.056294416" watchObservedRunningTime="2026-10-30 14:06:41.172428333 +0000 UTC m=+38.057186600" Oct 30 14:06:41.183666 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.182580 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv"] Oct 30 14:06:41.183666 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.182708 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv" Oct 30 14:06:41.257942 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.257908 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6pbv\" (UniqueName: \"kubernetes.io/projected/2a0b0e80-8291-4adf-83a3-7732f79e0f80-kube-api-access-p6pbv\") pod \"migrator-66df5d496b-42qwv\" (UID: \"2a0b0e80-8291-4adf-83a3-7732f79e0f80\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv" Oct 30 14:06:41.279813 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.279780 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4szpv\"" Oct 30 14:06:41.280768 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.280709 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 14:06:41.280912 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.280771 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:41.358976 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.358935 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:41.359132 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.359011 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:41.359132 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.359070 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6pbv\" (UniqueName: \"kubernetes.io/projected/2a0b0e80-8291-4adf-83a3-7732f79e0f80-kube-api-access-p6pbv\") pod \"migrator-66df5d496b-42qwv\" (UID: \"2a0b0e80-8291-4adf-83a3-7732f79e0f80\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv" Oct 30 14:06:41.359132 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.359086 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 14:06:41.359272 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.359136 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:41.359272 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.359158 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.359139345 +0000 UTC m=+46.243897604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : secret "router-metrics-certs-default" not found Oct 30 14:06:41.359272 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.359223 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:41.359272 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.359273 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls podName:08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.359258279 +0000 UTC m=+46.244016540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rt4nm" (UID: "08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65") : secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:41.359499 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.359350 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.359341726 +0000 UTC m=+46.244100007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : configmap references non-existent config key: service-ca.crt Oct 30 14:06:41.387219 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.387159 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6pbv\" (UniqueName: \"kubernetes.io/projected/2a0b0e80-8291-4adf-83a3-7732f79e0f80-kube-api-access-p6pbv\") pod \"migrator-66df5d496b-42qwv\" (UID: \"2a0b0e80-8291-4adf-83a3-7732f79e0f80\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv" Oct 30 14:06:41.459743 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.459698 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:41.459919 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.459759 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:41.459919 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.459831 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:41.459919 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.459867 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 14:06:41.459919 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.459876 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:41.460128 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.459926 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls podName:4070c760-6728-4166-a333-77af48851fd8 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.459912392 +0000 UTC m=+46.344670636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zrb4p" (UID: "4070c760-6728-4166-a333-77af48851fd8") : secret "samples-operator-tls" not found Oct 30 14:06:41.460128 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.460010 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:41.460128 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.460023 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fff9cc46-btf9d: secret "image-registry-tls" not found Oct 30 14:06:41.460128 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.460070 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls podName:bd9fc616-08fa-480b-bd77-622e4dfbf59e nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.460054446 +0000 UTC m=+46.344812710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls") pod "image-registry-8fff9cc46-btf9d" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e") : secret "image-registry-tls" not found Oct 30 14:06:41.460128 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.460120 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 14:06:41.460380 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.460150 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:43.460139822 +0000 UTC m=+40.344898065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : secret "insights-runtime-extractor-tls" not found Oct 30 14:06:41.460380 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.460196 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 14:06:41.460380 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.460229 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:43.460219291 +0000 UTC m=+40.344977539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : configmap "kube-rbac-proxy" not found Oct 30 14:06:41.512051 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.512018 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv" Oct 30 14:06:41.560983 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.560946 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:41.561153 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.561139 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:41.561226 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.561181 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:41.561287 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.561276 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:41.561287 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.561288 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89594b78c-qxvnh: secret "image-registry-tls" not found Oct 30 14:06:41.561363 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.561330 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls podName:fd07a434-a5f6-4cda-a93d-8824978aa590 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.561318696 +0000 UTC m=+46.446076953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls") pod "image-registry-89594b78c-qxvnh" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590") : secret "image-registry-tls" not found Oct 30 14:06:41.561753 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.561603 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 14:06:41.561753 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.561646 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 14:06:41.561753 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.561686 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert podName:c2536dd8-0f2c-4571-b0db-484894b2a99f nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.561675414 +0000 UTC m=+46.446433671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9s2h" (UID: "c2536dd8-0f2c-4571-b0db-484894b2a99f") : secret "networking-console-plugin-cert" not found Oct 30 14:06:41.561753 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:41.561740 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls podName:17aed83f-3fbe-4b2e-8714-e61122539496 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.561703684 +0000 UTC m=+46.446461941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls") pod "dns-default-fvkvc" (UID: "17aed83f-3fbe-4b2e-8714-e61122539496") : secret "dns-default-metrics-tls" not found Oct 30 14:06:41.844213 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:41.844183 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv"] Oct 30 14:06:41.848920 ip-10-0-129-158 kubenswrapper[2584]: W1030 14:06:41.848895 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0b0e80_8291_4adf_83a3_7732f79e0f80.slice/crio-4c77829b61a63e7b3ca4e30adf669c8548309e83fb294025b95b5f22116da7cf WatchSource:0}: Error finding container 4c77829b61a63e7b3ca4e30adf669c8548309e83fb294025b95b5f22116da7cf: Status 404 returned error can't find the container with id 4c77829b61a63e7b3ca4e30adf669c8548309e83fb294025b95b5f22116da7cf Oct 30 14:06:42.140961 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:42.140932 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv" event={"ID":"2a0b0e80-8291-4adf-83a3-7732f79e0f80","Type":"ContainerStarted","Data":"4c77829b61a63e7b3ca4e30adf669c8548309e83fb294025b95b5f22116da7cf"} Oct 30 14:06:43.146408 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:43.146370 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-stdbh" event={"ID":"aea273bc-03a8-41ad-8ed0-c1eab9a23a45","Type":"ContainerStarted","Data":"ee99f367cc3a55d18315b597015d0ccc70e3030937e68694a19acb4088b7baee"} Oct 30 14:06:43.146882 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:43.146487 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:43.189408 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:43.189354 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-stdbh" podStartSLOduration=35.323097816 podStartE2EDuration="39.189339866s" totalImagesPullingTime="3.86624205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:43.189237236 +0000 UTC m=+40.073995504" watchObservedRunningTime="2026-10-30 14:06:43.189339866 +0000 UTC m=+40.074098132" Oct 30 14:06:43.481463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:43.481381 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:43.481463 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:43.481439 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:43.481665 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:43.481543 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 14:06:43.481665 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:43.481577 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 14:06:43.481665 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:43.481637 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.48162077 +0000 UTC m=+44.366379027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : secret "insights-runtime-extractor-tls" not found Oct 30 14:06:43.481789 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:43.481668 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.481648463 +0000 UTC m=+44.366406721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : configmap "kube-rbac-proxy" not found Oct 30 14:06:43.889911 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:43.889886 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:43.890038 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:43.889939 2584 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:43.890237 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:43.890226 2584 scope.go:122] "RemoveContainer" containerID="6f398efb0490e4b545d0c2f056c5615256be15aa38b06eb15dad2a77ff08d592" Oct 30 14:06:43.890398 ip-10-0-129-158 kubenswrapper[2584]: E1030 14:06:43.890380 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-84mcb_openshift-console-operator(66b11a57-8746-4a93-b726-a877c2a22a98)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" podUID="66b11a57-8746-4a93-b726-a877c2a22a98" Oct 30 14:06:44.154620 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:44.154546 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vlmdd" event={"ID":"f6ae2497-ab63-4964-940a-e870d9131ae3","Type":"ContainerStarted","Data":"36a5e71a82512fde91cd91c9477fc6c729f509b9d44f298f565653c62fdb626f"} Oct 30 14:06:44.156822 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:44.156791 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv" event={"ID":"2a0b0e80-8291-4adf-83a3-7732f79e0f80","Type":"ContainerStarted","Data":"36f735787cf3e568fbaa3fdfc5a5d1458e65e3468418de3ac44e3ff1c9f1b50a"} Oct 30 14:06:44.194866 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:44.194814 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vlmdd" podStartSLOduration=35.046838674 podStartE2EDuration="41.194800752s" totalImagesPullingTime="6.147962078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:44.194104959 +0000 UTC m=+41.078863225" watchObservedRunningTime="2026-10-30 14:06:44.194800752 +0000 UTC m=+41.079559017" Oct 30 14:06:44.235457 ip-10-0-129-158 kubenswrapper[2584]: I1030 14:06:44.235405 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv" podStartSLOduration=1.859458858 podStartE2EDuration="3.235392186s" totalImagesPullingTime="1.375933328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:44.233892248 +0000 UTC m=+41.118650513" watchObservedRunningTime="2026-10-30 14:06:44.235392186 +0000 UTC m=+41.120150452" Oct 30 14:06:44.313865 ip-10-0-129-158 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 14:06:44.333145 ip-10-0-129-158 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 14:06:44.333360 ip-10-0-129-158 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 14:06:44.333422 ip-10-0-129-158 systemd[1]: kubelet.service: Consumed 2.968s CPU time. Oct 30 14:06:44.344498 ip-10-0-129-158 systemd[1]: Starting Kubernetes Kubelet... Oct 30 14:06:44.433299 ip-10-0-129-158 kubenswrapper[4968]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:06:44.433299 ip-10-0-129-158 kubenswrapper[4968]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 14:06:44.433299 ip-10-0-129-158 kubenswrapper[4968]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:06:44.433299 ip-10-0-129-158 kubenswrapper[4968]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:06:44.435252 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435174 4968 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:44.435252 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435253 4968 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435257 4968 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435260 4968 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435262 4968 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435265 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435268 4968 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435271 4968 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435273 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435276 4968 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435278 4968 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435281 4968 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435284 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435286 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435289 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435292 4968 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435294 4968 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435297 4968 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435300 4968 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435303 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435305 4968 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:44.435323 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435308 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435310 4968 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435313 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435316 4968 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435318 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435328 4968 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435331 4968 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435334 4968 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435337 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435340 4968 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435343 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435346 4968 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435348 4968 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435351 4968 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435353 4968 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435356 4968 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435359 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435361 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435364 4968 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:44.435833 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435366 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435369 4968 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435371 4968 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435374 4968 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435376 4968 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435379 4968 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435382 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435384 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435387 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435390 4968 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435393 4968 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435395 4968 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435398 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435401 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435404 4968 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435406 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435409 4968 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435412 4968 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435414 4968 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:44.436289 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435419 4968 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435423 4968 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435427 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435430 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435433 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435435 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435438 4968 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435441 4968 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435443 4968 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435446 4968 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435449 4968 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435451 4968 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435454 4968 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435456 4968 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435459 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435462 4968 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435464 4968 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435468 4968 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435470 4968 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435473 4968 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:44.436781 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435477 4968 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435480 4968 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435482 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435487 4968 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435491 4968 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435494 4968 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435498 4968 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435501 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435503 4968 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435506 4968 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435509 4968 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435512 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435514 4968 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435520 4968 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435522 4968 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435525 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435528 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435531 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435534 4968 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:44.437270 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435537 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435539 4968 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435542 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435545 4968 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435547 4968 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435549 4968 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435552 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.435555 4968 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436304 4968 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436311 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436314 4968 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436317 4968 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436320 4968 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436323 4968 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436326 4968 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436328 4968 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436331 4968 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436334 4968 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436337 4968 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436339 4968 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:44.437748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436342 4968 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436345 4968 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436347 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436350 4968 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436353 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436355 4968 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436358 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436361 4968 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436364 4968 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436367 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436369 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436372 4968 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436374 4968 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436377 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436379 4968 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436382 4968 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436384 4968 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436387 4968 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436390 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:44.438244 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436393 4968 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436396 4968 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436400 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436404 4968 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436407 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436410 4968 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436413 4968 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436415 4968 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436419 4968 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436424 4968 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436427 4968 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436429 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436432 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436435 4968 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436438 4968 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436440 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436443 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436445 4968 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436448 4968 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:44.438757 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436450 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436453 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436456 4968 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436459 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436461 4968 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436464 4968 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436466 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436469 4968 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436471 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436475 4968 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436478 4968 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436481 4968 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436483 4968 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436486 4968 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436489 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436492 4968 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436494 4968 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436497 4968 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436499 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436502 4968 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:44.439236 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436505 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436508 4968 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436510 4968 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436513 4968 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436515 4968 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436518 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436520 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436523 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436525 4968 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436528 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436530 4968 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436533 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436535 4968 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436538 4968 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436540 4968 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436546 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436549 4968 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436552 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436554 4968 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:44.439769 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436557 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436560 4968 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436562 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436565 4968 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436567 4968 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436573 4968 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436575 4968 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436578 4968 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436580 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436583 4968 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436585 4968 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436588 4968 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436591 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436594 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436597 4968 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436600 4968 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.436603 4968 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437161 4968 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437166 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:44.440241 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437169 4968 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437172 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437175 4968 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437177 4968 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437180 4968 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437183 4968 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437186 4968 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437188 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437191 4968 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437199 4968 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437202 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437204 4968 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437207 4968 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437210 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437212 4968 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437215 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437217 4968 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437220 4968 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437224 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:44.440697 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437228 4968 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437230 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437233 4968 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437235 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437238 4968 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437241 4968 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437243 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437246 4968 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437249 4968 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437251 4968 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437254 4968 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437256 4968 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437259 4968 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437262 4968 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437264 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437267 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437269 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437272 4968 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437275 4968 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437278 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:44.441193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437281 4968 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437283 4968 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437286 4968 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437294 4968 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437297 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437300 4968 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437302 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437305 4968 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437308 4968 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437310 4968 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437313 4968 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437319 4968 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437322 4968 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437325 4968 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437327 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437330 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437333 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437335 4968 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437338 4968 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437340 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:44.441673 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437343 4968 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437346 4968 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437349 4968 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437351 4968 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437354 4968 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437356 4968 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437359 4968 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437363 4968 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437367 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437371 4968 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437374 4968 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437377 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437379 4968 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437382 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437385 4968 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437387 4968 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437396 4968 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437400 4968 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437404 4968 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:44.442170 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437407 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437410 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437414 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437417 4968 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437421 4968 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437423 4968 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437426 4968 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437428 4968 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437431 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437433 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437436 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437438 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437441 4968 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437444 4968 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437446 4968 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437449 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437451 4968 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437454 4968 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437456 4968 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:44.442626 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437459 4968 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437462 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437464 4968 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437467 4968 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437470 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437473 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.437475 4968 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437575 4968 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437585 4968 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437600 4968 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437605 4968 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437616 4968 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437619 4968 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437624 4968 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437628 4968 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437633 4968 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437636 4968 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437640 4968 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437643 4968 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437647 4968 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437650 4968 flags.go:64] FLAG: --cgroup-root="" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437653 4968 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437656 4968 flags.go:64] FLAG: --client-ca-file="" Oct 30 14:06:44.443121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437659 4968 flags.go:64] FLAG: --cloud-provider="external" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437662 4968 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437669 4968 flags.go:64] FLAG: --cluster-domain="" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437672 4968 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437675 4968 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437678 4968 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437681 4968 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437685 4968 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437688 4968 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437691 4968 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437695 4968 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437698 4968 flags.go:64] FLAG: --contention-profiling="false" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437701 4968 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437704 4968 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437707 4968 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437710 4968 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437726 4968 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437729 4968 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437732 4968 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437734 4968 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437737 4968 flags.go:64] FLAG: --enable-server="true" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437748 4968 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437755 4968 flags.go:64] FLAG: --event-burst="100" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437758 4968 flags.go:64] FLAG: --event-qps="50" Oct 30 14:06:44.443672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437764 4968 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437767 4968 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437771 4968 flags.go:64] FLAG: --eviction-hard="" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437775 4968 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437778 4968 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437781 4968 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437784 4968 flags.go:64] FLAG: --eviction-soft="" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437787 4968 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437791 4968 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437794 4968 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437797 4968 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437800 4968 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437803 4968 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437806 4968 flags.go:64] FLAG: --feature-gates="" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437813 4968 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437816 4968 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437819 4968 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437822 4968 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437825 4968 flags.go:64] FLAG: --healthz-port="10248" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437829 4968 flags.go:64] FLAG: --help="false" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437831 4968 flags.go:64] FLAG: --hostname-override="ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437835 4968 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437838 4968 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 14:06:44.444305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437840 4968 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437844 4968 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437847 4968 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437850 4968 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437853 4968 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437856 4968 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437859 4968 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437868 4968 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437871 4968 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437877 4968 flags.go:64] FLAG: --kube-reserved="" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437880 4968 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437884 4968 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437887 4968 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437890 4968 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437892 4968 flags.go:64] FLAG: --lock-file="" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437895 4968 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437898 4968 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437901 4968 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437907 4968 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437910 4968 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437913 4968 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437916 4968 flags.go:64] FLAG: --logging-format="text" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437919 4968 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437922 4968 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 14:06:44.445090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437925 4968 flags.go:64] FLAG: --manifest-url="" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437928 4968 flags.go:64] FLAG: --manifest-url-header="" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437932 4968 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437935 4968 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437939 4968 flags.go:64] FLAG: --max-pods="110" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437942 4968 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437945 4968 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437948 4968 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437951 4968 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437954 4968 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437957 4968 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437960 4968 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437967 4968 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437970 4968 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437973 4968 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437976 4968 flags.go:64] FLAG: --pod-cidr="" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437985 4968 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437990 4968 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437993 4968 flags.go:64] FLAG: --pods-per-core="0" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.437997 4968 flags.go:64] FLAG: --port="10250" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438001 4968 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438004 4968 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dc2660bf87d08fc7" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438007 4968 flags.go:64] FLAG: --qos-reserved="" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438010 4968 flags.go:64] FLAG: --read-only-port="10255" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438013 4968 flags.go:64] FLAG: --register-node="true" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438015 4968 flags.go:64] FLAG: --register-with-taints="" Oct 30 14:06:44.446096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438019 4968 flags.go:64] FLAG: --registry-burst="10" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438022 4968 flags.go:64] FLAG: --registry-qps="5" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438025 4968 flags.go:64] FLAG: --reserved-cpus="" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438028 4968 flags.go:64] FLAG: --reserved-memory="" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438032 4968 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438035 4968 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438038 4968 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438041 4968 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438044 4968 flags.go:64] FLAG: --runonce="false" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438047 4968 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438050 4968 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438053 4968 flags.go:64] FLAG: --seccomp-default="false" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438056 4968 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438059 4968 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438062 4968 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438066 4968 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438069 4968 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438072 4968 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438074 4968 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438077 4968 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438080 4968 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438083 4968 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438086 4968 flags.go:64] FLAG: --system-cgroups="" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438096 4968 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438102 4968 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 14:06:44.446806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438106 4968 flags.go:64] FLAG: --tls-cert-file="" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438109 4968 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438116 4968 flags.go:64] FLAG: --tls-min-version="" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438119 4968 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438122 4968 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438125 4968 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438128 4968 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438131 4968 flags.go:64] FLAG: --v="2" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438135 4968 flags.go:64] FLAG: --version="false" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438139 4968 flags.go:64] FLAG: --vmodule="" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438143 4968 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438147 4968 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438337 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438341 4968 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438344 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438347 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438350 4968 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438352 4968 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438355 4968 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438357 4968 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438360 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438363 4968 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438365 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:44.447486 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438368 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438370 4968 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438373 4968 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438376 4968 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438378 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438381 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438386 4968 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438391 4968 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438400 4968 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438405 4968 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438408 4968 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438411 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438413 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438416 4968 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438419 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438422 4968 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438424 4968 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438427 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438430 4968 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:44.448098 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438432 4968 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438435 4968 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438438 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438440 4968 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438443 4968 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438445 4968 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438448 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438450 4968 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438453 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438455 4968 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438458 4968 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438461 4968 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438464 4968 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438466 4968 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438469 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438471 4968 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438474 4968 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438477 4968 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438479 4968 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:44.448565 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438482 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438485 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438487 4968 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438497 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438500 4968 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438502 4968 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438505 4968 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438508 4968 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438510 4968 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438512 4968 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438515 4968 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438518 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438520 4968 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438523 4968 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438526 4968 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438528 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438531 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438534 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438536 4968 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:44.449140 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438539 4968 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438542 4968 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438546 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438549 4968 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438552 4968 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438555 4968 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438557 4968 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438560 4968 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438562 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438565 4968 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438567 4968 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438570 4968 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438572 4968 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438575 4968 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438578 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438580 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438584 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438593 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438596 4968 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:44.449823 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438599 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438601 4968 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438604 4968 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438606 4968 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438609 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438612 4968 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438614 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438617 4968 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438620 4968 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438622 4968 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438625 4968 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438627 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438630 4968 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438632 4968 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438635 4968 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438638 4968 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438642 4968 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438646 4968 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:44.450473 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438650 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.438686 4968 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438748 4968 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438753 4968 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438756 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438759 4968 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438761 4968 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438764 4968 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438767 4968 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438770 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438772 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438775 4968 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438778 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438791 4968 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438794 4968 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438797 4968 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:44.451074 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438799 4968 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438802 4968 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438805 4968 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438807 4968 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438810 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438812 4968 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438815 4968 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438818 4968 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438820 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438823 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438827 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438830 4968 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438832 4968 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438836 4968 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438839 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438842 4968 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438844 4968 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438847 4968 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438849 4968 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438852 4968 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:44.451483 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438855 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438857 4968 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438860 4968 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438862 4968 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438865 4968 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438867 4968 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438870 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438872 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438875 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438878 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438886 4968 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438889 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438892 4968 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438895 4968 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438897 4968 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438900 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438902 4968 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438904 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438908 4968 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:44.452113 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438910 4968 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438913 4968 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438915 4968 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438919 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438922 4968 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438924 4968 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438927 4968 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438929 4968 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438932 4968 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438934 4968 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438937 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438940 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438942 4968 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438944 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438947 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438949 4968 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438952 4968 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438955 4968 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438957 4968 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438960 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:44.452795 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438963 4968 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438965 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438968 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438970 4968 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438985 4968 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438988 4968 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438991 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438993 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.438998 4968 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439001 4968 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439004 4968 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439008 4968 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439010 4968 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439013 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439015 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439019 4968 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439022 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439025 4968 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439028 4968 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:44.453354 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439030 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439032 4968 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439035 4968 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439038 4968 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439040 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439043 4968 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439046 4968 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439048 4968 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439051 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439053 4968 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439056 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439058 4968 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439061 4968 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.439064 4968 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.443251 4968 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.443270 4968 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.443287 4968 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.443298 4968 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443382 4968 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443388 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:44.453842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443392 4968 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443395 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443398 4968 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443401 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443403 4968 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443413 4968 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443417 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443420 4968 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443423 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443425 4968 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443428 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443431 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443433 4968 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443436 4968 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443438 4968 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443441 4968 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443444 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443447 4968 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443449 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:44.454322 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443452 4968 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443456 4968 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443460 4968 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443463 4968 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443466 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443470 4968 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443474 4968 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443477 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443479 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443482 4968 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443485 4968 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443497 4968 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443500 4968 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443503 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443506 4968 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443508 4968 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443511 4968 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443514 4968 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443516 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:44.454792 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443519 4968 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443522 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443524 4968 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443527 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443529 4968 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443532 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443535 4968 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443537 4968 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443540 4968 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443542 4968 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443545 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443548 4968 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443551 4968 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443553 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443556 4968 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443558 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443561 4968 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443564 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443567 4968 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443570 4968 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:44.455249 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443572 4968 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443575 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443577 4968 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443580 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443583 4968 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443591 4968 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443594 4968 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443597 4968 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443599 4968 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443602 4968 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443604 4968 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443607 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443609 4968 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443612 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443615 4968 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443617 4968 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443620 4968 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443622 4968 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443625 4968 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:44.455829 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443628 4968 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443630 4968 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443633 4968 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443635 4968 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443638 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443641 4968 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443643 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443646 4968 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443648 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443651 4968 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443653 4968 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443656 4968 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443659 4968 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443661 4968 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443664 4968 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443666 4968 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443669 4968 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443671 4968 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443674 4968 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443682 4968 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:44.456312 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443685 4968 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443688 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443691 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443694 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443696 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443699 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443701 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.443741 4968 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443950 4968 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443956 4968 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443960 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443963 4968 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443966 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443968 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443971 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:44.456806 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443974 4968 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443976 4968 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443979 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443982 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443984 4968 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443987 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443989 4968 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443992 4968 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443995 4968 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.443998 4968 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444000 4968 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444003 4968 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444005 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444008 4968 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444010 4968 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444013 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444015 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444024 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444027 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444029 4968 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:44.457163 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444032 4968 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444034 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444037 4968 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444039 4968 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444042 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444045 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444047 4968 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444050 4968 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444052 4968 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444055 4968 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444057 4968 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444060 4968 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444062 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444065 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444067 4968 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444070 4968 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444073 4968 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444075 4968 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444078 4968 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:44.457650 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444080 4968 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444083 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444085 4968 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444088 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444090 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444093 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444095 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444098 4968 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444100 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444103 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444105 4968 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444113 4968 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444116 4968 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444119 4968 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444122 4968 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444124 4968 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444127 4968 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444129 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444132 4968 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:44.458156 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444134 4968 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444137 4968 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444139 4968 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444142 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444144 4968 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444147 4968 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444149 4968 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444152 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444155 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444157 4968 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444159 4968 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444162 4968 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444164 4968 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444167 4968 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444169 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444172 4968 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444174 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444177 4968 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444180 4968 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444182 4968 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:44.458614 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444185 4968 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444187 4968 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444195 4968 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444198 4968 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444201 4968 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444210 4968 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444216 4968 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444219 4968 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444221 4968 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444224 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444226 4968 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444229 4968 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444231 4968 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444234 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444237 4968 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444239 4968 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444242 4968 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444244 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444247 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444250 4968 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:44.459096 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444254 4968 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.444276 4968 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444323 4968 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444327 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444331 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444334 4968 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444337 4968 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444339 4968 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444342 4968 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444345 4968 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444347 4968 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444350 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444353 4968 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444355 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444358 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:06:44.459627 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444360 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444363 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444365 4968 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444373 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444378 4968 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444381 4968 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444383 4968 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444386 4968 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444389 4968 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444391 4968 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444394 4968 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444396 4968 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444399 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444401 4968 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444404 4968 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444406 4968 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444409 4968 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444412 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444415 4968 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444417 4968 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:06:44.460147 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444420 4968 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444422 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444425 4968 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444427 4968 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444430 4968 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444433 4968 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444435 4968 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444438 4968 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444440 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444443 4968 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444445 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444448 4968 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444451 4968 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444453 4968 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444455 4968 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444458 4968 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.444497 4968 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445025 4968 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445034 4968 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445039 4968 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:06:44.460641 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445044 4968 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445048 4968 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445053 4968 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445057 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445061 4968 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445067 4968 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445072 4968 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445076 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445087 4968 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445092 4968 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445096 4968 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445101 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445105 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445112 4968 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445119 4968 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445127 4968 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445136 4968 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445142 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445147 4968 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:06:44.461206 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445151 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445161 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445165 4968 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445170 4968 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445174 4968 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445178 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445182 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445187 4968 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445191 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445196 4968 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445239 4968 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445243 4968 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445247 4968 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445256 4968 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445261 4968 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445265 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445269 4968 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445273 4968 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445278 4968 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445282 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:06:44.462033 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445287 4968 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445291 4968 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445295 4968 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445299 4968 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445304 4968 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445308 4968 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445318 4968 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445322 4968 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445329 4968 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445334 4968 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445338 4968 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445342 4968 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445347 4968 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:44.445351 4968 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.445447 4968 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.446988 4968 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.447047 4968 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.447253 4968 server.go:1014] "Starting client certificate rotation" Oct 30 14:06:44.462748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.447401 4968 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 14:06:44.463516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.447452 4968 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 14:01:03 +0000 UTC" deadline="2028-04-19 02:21:01.773042192 +0000 UTC" Oct 30 14:06:44.463516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.447479 4968 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12876h14m17.325565103s" Oct 30 14:06:44.463516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.447686 4968 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:06:44.463516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.448737 4968 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:06:44.463516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.450373 4968 log.go:25] "Validated CRI v1 runtime API" Oct 30 14:06:44.463516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.452197 4968 log.go:25] "Validated CRI v1 image API" Oct 30 14:06:44.463516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.452948 4968 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 14:06:44.463516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.454829 4968 fs.go:135] Filesystem UUIDs: map[0058fbd1-3288-49a7-b7be-227d2a6d2706:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f3886125-b35a-4731-a84d-2baf6ef0f019:/dev/nvme0n1p4] Oct 30 14:06:44.464034 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.454848 4968 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bc892c923ac74be04bf036a21cf1b28787d5c72dba173b2e407434e9be65521/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bc892c923ac74be04bf036a21cf1b28787d5c72dba173b2e407434e9be65521/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10c25997c409f0b29d95aa9f4484707384b72e3296304e35c7599b29249c9a5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10c25997c409f0b29d95aa9f4484707384b72e3296304e35c7599b29249c9a5d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1804ec947805fc1806e9314ec207a3ffd212b9a4bc457bbd48a2ca0b28d02c2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1804ec947805fc1806e9314ec207a3ffd212b9a4bc457bbd48a2ca0b28d02c2a/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f40c0d7ae2c4ae218631ecef0226219b409c88f84c1cc8682c6fa9d2519ba61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f40c0d7ae2c4ae218631ecef0226219b409c88f84c1cc8682c6fa9d2519ba61/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4079ca19c89bdeaaf49de19f510759b96f2c4ba34a2b40ea6fcbe96b86afc2fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4079ca19c89bdeaaf49de19f510759b96f2c4ba34a2b40ea6fcbe96b86afc2fd/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c77829b61a63e7b3ca4e30adf669c8548309e83fb294025b95b5f22116da7cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c77829b61a63e7b3ca4e30adf669c8548309e83fb294025b95b5f22116da7cf/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cc6f99f8e3efe6cf211c29d73d271b4b53c9b3d5936ef0937345119ff3b05d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cc6f99f8e3efe6cf211c29d73d271b4b53c9b3d5936ef0937345119ff3b05d6/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51b981c69d8492be055c2dc1d8f7be2b52ce30979a988059582fc6a79a73b714/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51b981c69d8492be055c2dc1d8f7be2b52ce30979a988059582fc6a79a73b714/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52aef7cad4bd812500483db121cfae723767e4715bcea840fb8cdb3305c03e9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52aef7cad4bd812500483db121cfae723767e4715bcea840fb8cdb3305c03e9c/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/534e25f2af392926f0949ee3bb6b3799603647967a4fc967fe0e01ae8014d0dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/534e25f2af392926f0949ee3bb6b3799603647967a4fc967fe0e01ae8014d0dc/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e7a18c0a9391981fba99e22e260ba8b224ed58dc92119c0753c231079b552ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e7a18c0a9391981fba99e22e260ba8b224ed58dc92119c0753c231079b552ca/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75c66915c3d4f4e4d4c22877e594eb5c908adc64357a5dc45d1ff3dc30feff24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75c66915c3d4f4e4d4c22877e594eb5c908adc64357a5dc45d1ff3dc30feff24/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e3af484b38bfa7e08e90b016ba031ea62141d0b5e580f3e77d688b42155a0d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e3af484b38bfa7e08e90b016ba031ea62141d0b5e580f3e77d688b42155a0d1/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3db897157965152baeddf9496ee5d0dc0e65ea61064629d8ae00d218ac98cb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3db897157965152baeddf9496ee5d0dc0e65ea61064629d8ae00d218ac98cb1/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4d69a5c88e477db630b182c8faf58c2aa21014498416fa1d31968f939c475fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4d69a5c88e477db630b182c8faf58c2aa21014498416fa1d31968f939c475fd/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9ce8bbfd5ed5fcbbf19934621bfc94b2d35d455fb7ebf77f4b4c86bb21ae612/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9ce8bbfd5ed5fcbbf19934621bfc94b2d35d455fb7ebf77f4b4c86bb21ae612/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5e718a5497402d58bac6780c3a2e6c291eede489a5938ded3087c0e77797ddd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5e718a5497402d58bac6780c3a2e6c291eede489a5938ded3087c0e77797ddd/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b612015fb112ba2dbc4456abd28335ab8c9f6ec986d7e3f0991c77b02c7502eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b612015fb112ba2dbc4456abd28335ab8c9f6ec986d7e3f0991c77b02c7502eb/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd6bd9f6c7a2ec2513e595e95e160b238a68d219ae90dcf25d7b31dc58fe390e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd6bd9f6c7a2ec2513e595e95e160b238a68d219ae90dcf25d7b31dc58fe390e/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2b7943ae35026336881dcd417bae55e6e08df8ebcd6773537471075a3fa92d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2b7943ae35026336881dcd417bae55e6e08df8ebcd6773537471075a3fa92d9/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed8f58363bcee5fbb7ff703de95bf9882033f5112ca4510aa71c383aa4acbd6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed8f58363bcee5fbb7ff703de95bf9882033f5112ca4510aa71c383aa4acbd6f/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4dca8d9b5a5f58da6d765ffd9051266f97f624e74a03bf7ec76b713465e78e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4dca8d9b5a5f58da6d765ffd9051266f97f624e74a03bf7ec76b713465e78e2/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff67637528372b258ad2cc6dd8f09ed6aeedd40078fbacaf0659ec0c99e3f98e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff67637528372b258ad2cc6dd8f09ed6aeedd40078fbacaf0659ec0c99e3f98e/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03049b65-310d-4ad7-b614-1daf3a50b528/volumes/kubernetes.io~projected/kube-api-access-f62nk:{mountpoint:/var/lib/kubelet/pods/03049b65-310d-4ad7-b614-1daf3a50b528/volumes/kubernetes.io~projected/kube-api-access-f62nk major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65/volumes/kubernetes.io~projected/kube-api-access-7sqjv:{mountpoint:/var/lib/kubelet/pods/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65/volumes/kubernetes.io~projected/kube-api-access-7sqjv major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17aed83f-3fbe-4b2e-8714-e61122539496/volumes/kubernetes.io~projected/kube-api-access-82bmb:{mountpoint:/var/lib/kubelet/pods/17aed83f-3fbe-4b2e-8714-e61122539496/volumes/kubernetes.io~projected/kube-api-access-82bmb major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cdffa3f-7378-4f50-b1bc-e87aa647c4d8/volumes/kubernetes.io~projected/kube-api-access-tbcd9:{mountpoint:/var/lib/kubelet/pods/1cdffa3f-7378-4f50-b1bc-e87aa647c4d8/volumes/kubernetes.io~projected/kube-api-access-tbcd9 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a0b0e80-8291-4adf-83a3-7732f79e0f80/volumes/kubernetes.io~projected/kube-api-access-p6pbv:{mountpoint:/var/lib/kubelet/pods/2a0b0e80-8291-4adf-83a3-7732f79e0f80/volumes/kubernetes.io~projected/kube-api-access-p6pbv major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e3b429d-b5ef-40bf-b31a-9bb2b5085010/volumes/kubernetes.io~projected/kube-api-access-gqwzb:{mountpoint:/var/lib/kubelet/pods/2e3b429d-b5ef-40bf-b31a-9bb2b5085010/volumes/kubernetes.io~projected/kube-api-access-gqwzb major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/354da4d4-65ce-4f03-a9aa-f7840bf28711/volumes/kubernetes.io~projected/kube-api-access-bcd7x:{mountpoint:/var/lib/kubelet/pods/354da4d4-65ce-4f03-a9aa-f7840bf28711/volumes/kubernetes.io~projected/kube-api-access-bcd7x major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3829121e-3790-47ff-b827-8774a70fec04/volumes/kubernetes.io~projected/kube-api-access-7w7rf:{mountpoint:/var/lib/kubelet/pods/3829121e-3790-47ff-b827-8774a70fec04/volumes/kubernetes.io~projected/kube-api-access-7w7rf major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a8ec17f-ff03-4af2-8885-1707bf84ec85/volumes/kubernetes.io~projected/kube-api-access-8qdqj:{mountpoint:/var/lib/kubelet/pods/3a8ec17f-ff03-4af2-8885-1707bf84ec85/volumes/kubernetes.io~projected/kube-api-access-8qdqj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4070c760-6728-4166-a333-77af48851fd8/volumes/kubernetes.io~projected/kube-api-access-thjb5:{mountpoint:/var/lib/kubelet/pods/4070c760-6728-4166-a333-77af48851fd8/volumes/kubernetes.io~projected/kube-api-access-thjb5 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4171ff7a-5894-49d1-8683-606723fccdf9/volumes/kubernetes.io~projected/kube-api-access-ssmtf:{mountpoint:/var/lib/kubelet/pods/4171ff7a-5894-49d1-8683-606723fccdf9/volumes/kubernetes.io~projected/kube-api-access-ssmtf major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4171ff7a-5894-49d1-8683-606723fccdf9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4171ff7a-5894-49d1-8683-606723fccdf9/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42511697-bcd4-4df3-819c-ab38ba5159c3/volumes/kubernetes.io~projected/kube-api-access-rrrj9:{mountpoint:/var/lib/kubelet/pods/42511697-bcd4-4df3-819c-ab38ba5159c3/volumes/kubernetes.io~projected/kube-api-access-rrrj9 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42511697-bcd4-4df3-819c-ab38ba5159c3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/42511697-bcd4-4df3-819c-ab38ba5159c3/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f07e86b-0018-4c02-a9c6-cc24bfecd068/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4f07e86b-0018-4c02-a9c6-cc24bfecd068/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f07e86b-0018-4c02-a9c6-cc24bfecd068/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4f07e86b-0018-4c02-a9c6-cc24bfecd068/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f07e86b-0018-4c02-a9c6-cc24bfecd068/volumes/kubernetes.io~projected/kube-api-access-wmq8x:{mountpoint:/var/lib/kubelet/pods/4f07e86b-0018-4c02-a9c6-cc24bfecd068/volumes/kubernetes.io~projected/kube-api-access-wmq8x major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b388277-9bf2-45be-b85d-2eafd1a1c543/volumes/kubernetes.io~projected/kube-api-access-5xhlw:{mountpoint:/var/lib/kubelet/pods/5b388277-9bf2-45be-b85d-2eafd1a1c543/volumes/kubernetes.io~projected/kube-api-access-5xhlw major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66b11a57-8746-4a93-b726-a877c2a22a98/volumes/kubernetes.io~projected/kube-api-access-l2zxn:{mountpoint:/var/lib/kubelet/pods/66b11a57-8746-4a93-b726-a877c2a22a98/volumes/kubernetes.io~projected/kube-api-access-l2zxn major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66b11a57-8746-4a93-b726-a877c2a22a98/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/66b11a57-8746-4a93-b726-a877c2a22a98/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70ab068a-ed6b-4fae-8019-2e7c5873ab43/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/70ab068a-ed6b-4fae-8019-2e7c5873ab43/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70ab068a-ed6b-4fae-8019-2e7c5873ab43/volumes/kubernetes.io~projected/kube-api-access-9pwr6:{mountpoint:/var/lib/kubelet/pods/70ab068a-ed6b-4fae-8019-2e7c5873ab43/volumes/kubernetes.io~projected/kube-api-access-9pwr6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70ab068a-ed6b-4fae-8019-2e7c5873ab43/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/70ab068a-ed6b-4fae-8019-2e7c5873ab43/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78841e01-5bf7-4348-9802-09a81adeb88a/volumes/kubernetes.io~projected/kube-api-access-xjm94:{mountpoint:/var/lib/kubelet/pods/78841e01-5bf7-4348-9802-09a81adeb88a/volumes/kubernetes.io~projected/kube-api-access-xjm94 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81e6ce71-7fb4-40ab-b14a-54457b97bd6a/volumes/kubernetes.io~projected/kube-api-access-cgn77:{mountpoint:/var/lib/kubelet/pods/81e6ce71-7fb4-40ab-b14a-54457b97bd6a/volumes/kubernetes.io~projected/kube-api-access-cgn77 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/911b39fc-f2e6-4094-9be0-68606d2aa4f1/volumes/kubernetes.io~projected/kube-api-access-5vklh:{mountpoint:/var/lib/kubelet/pods/911b39fc-f2e6-4094-9be0-68606d2aa4f1/volumes/kubernetes.io~projected/kube-api-access-5vklh major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/911b39fc-f2e6-4094-9be0-68606d2aa4f1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/911b39fc-f2e6-4094-9be0-68606d2aa4f1/volumes/kubernetes.io~secret/cert major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4/volumes/kubernetes.io~projected/kube-api-access-9gjlh:{mountpoint:/var/lib/kubelet/pods/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4/volumes/kubernetes.io~projected/kube-api-access-9gjlh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5/volumes/kubernetes.io~projected/kube-api-access-56lth:{mountpoint:/var/lib/kubelet/pods/a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5/volumes/kubernetes.io~projected/kube-api-access-56lth major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aea273bc-03a8-41ad-8ed0-c1eab9a23a45/volumes/kubernetes.io~projected/kube-api-access-f7vhn:{mountpoint:/var/lib/kubelet/pods/aea273bc-03a8-41ad-8ed0-c1eab9a23a45/volumes/kubernetes.io~projected/kube-api-access-f7vhn major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~projected/kube-api-access-hxk4p:{mountpoint:/var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~projected/kube-api-access-hxk4p major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfc6f25a-6a7a-472c-af78-683613ffb7a5/volumes/kubernetes.io~projected/kube-api-access-77kg8:{mountpoint:/var/lib/kubelet/pods/dfc6f25a-6a7a-472c-af78-683613ffb7a5/volumes/kubernetes.io~projected/kube-api-access-77kg8 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfc6f25a-6a7a-472c-af78-683613ffb7a5/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/dfc6f25a-6a7a-472c-af78-683613ffb7a5/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfc6f25a-6a7a-472c-af78-683613ffb7a5/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/dfc6f25a-6a7a-472c-af78-683613ffb7a5/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ead802d8-7ee2-4381-ad65-723736aabd12/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ead802d8-7ee2-4381-ad65-723736aabd12/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6ae2497-ab63-4964-940a-e870d9131ae3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f6ae2497-ab63-4964-940a-e870d9131ae3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41/volumes/kubernetes.io~projected/kube-api-access-zw84x:{mountpoint:/var/lib/kubelet/pods/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41/volumes/kubernetes.io~projected/kube-api-access-zw84x major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb9d3ab7-8a7f-4ccd-a336-912b00332110/volumes/kubernetes.io~projected/kube-api-access-928ww:{mountpoint:/var/lib/kubelet/pods/fb9d3ab7-8a7f-4ccd-a336-912b00332110/volumes/kubernetes.io~projected/kube-api-access-928ww major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb9d3ab7-8a7f-4ccd-a336-912b00332110/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fb9d3ab7-8a7f-4ccd-a336-912b00332110/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~projected/kube-api-access-jv27n:{mountpoint:/var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~projected/kube-api-access-jv27n major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe78a7b6-cba1-4c52-b190-0febeaab5035/volumes/kubernetes.io~projected/kube-api-access-lztlh:{mountpoint:/var/lib/kubelet/pods/fe78a7b6-cba1-4c52-b190-0febeaab5035/volumes/kubernetes.io~projected/kube-api-access-lztlh major:0 minor:237 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/8752cef61be3e02904d71fc4bf198585ff5f45387079d441fb1912dcbe31f162/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7158fed6bfca7ff7119d1d21049c6b8cc2fba2d0cd2f5ad689702827e5e690e8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0ed767eff37fb6a974739efacd44a4ff2438c7a8b349f976d4a5978326f82069/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/b7fec3a7b1e881473f3bf052fed3b7909ea6845edeb6d28117852d4775e43efd/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/dba0b7b32cddb57bdf53151790c065247453839ee0e982a5e2f17b493593d85c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ca1cb343e839dbe200363e383544b4945d1d5bdb67021ad287794138e8656e6e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5f994733a92026b465bd588ca6f0c60ad114c5c60b602facff795610a2638096/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/97cfddde2d2f9aa084026ea77b1dd384fe421929f49690c6c8812a3aee2aa9be/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d15c08bd6f8905a4ae37b799ec2262e9168b6e9e9ebc212f5fb23feaafa40308/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/79e3aa3230e75044059efa6a8c7012726866e3b9010441b4c149a06aa59ba232/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/e1628d470e550f6759d449a015dc54610b168e485daccf7ad837695979c632cc/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/c83d6925e083c51e59da3755fac60670ea3174e3729a485104046f3f89c28eb1/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/f3da23237cbf0e4cbc1ff98694e4f52d481ac1f890ad13bde6e610cbbcbcddb7/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/85680ba69b3bd0f587022b91e1608968374a6e4e7f25c6502cc1fc36d2c71fa4/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7dd870afcf7c511aa598fc4db109ed04f2b6e4f5c04f14ce453f873b2f239451/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6143d5f445d3492b506f6dff4e5863261381051c3ac4aef80bef6cb4c34419bd/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/61ea3e486199b3e3de0d7de41da659eb6fa7f8efc1ce2a06d6d62c3d80f741bf/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a0eced34bcd1bac5c12cfea0bfde87fa0f249fa8aa0c9e0574c7c9738436785d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5ea50b6d1fca00f35640d8f9ff4c558713f1d85ee66d04fbed0c4470eda2d556/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0cf79110fe6b1efbc3c03127dbf2b0193872e9f3875953491e24cd4df1a73df0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9e3415969ad3ff795d1bc1d80d2082627c40e55e75b10d581bc055926b9421df/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/e12a3fbf2d3c1e9f1bb1fe49d79c8a1227a00b7cdcd9e09187def2efbb2ffd19/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/bca0e0facaf7a4bdfaf3e4f3af9af39f4029a12a3409e325909f151a78d4f767/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/f08190ac76bde77805283c99a8af74e89ee47ba7666ddac7e5995d3792e2d357/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/0000cc5d398cec6c5fa9540d0e07f58a44c7e53f8d3e189deafb561a07aacf6f/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/faa71c971bf90decd9d8c1b06ce8a07e206ae536a50e5943c72cb9e1a73c9498/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/b5a8457636b6ecf31114e5967028714a1329de7e215d0a3dca082b8cc8ba0811/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/a34f432ca1a282a35e2cc196adbed30bcef8d3df6edc5298002d8ae3a20e37ed/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/79b1e5660c47d8bb769d442edd0ed0b6926ec7ff7391a278a0168a60846c42a0/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/545c38b108a36d1b1970325d32bc2e5f635ebbbde1942b4ffc28fe7f23bb1388/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/ba732139c4f5927fad230408822e0d4c681c5a91f7b6a4ecdd6c4e0e50c73e55/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/2bdd48508cf644769bf94929ed9a8af32f053dc7938b137c5a98e26df3d8cd67/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/c3287d5d7685acf3d7b821915da68ea41c837da6dec1463aee347cd6e16078a6/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/8f36cce6f5d112314ede48a32772e220b1def848565c9ef7dde9f16a4180a4c7/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/882e8276c50977758e77b943537cf80f229201ad4cc4b54bc5a78c1901e9a4dd/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/e1ec198c1eb9b19ba5ef85e8657495b9f52a155f1288675807d78e406d02b97a/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/4a114cef6ed17fbb4ed87cb1c637b68a569954450a50010ad68709f4b6573bd6/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/98a06016032b6fcc0c85480d2b6a9f73393e9e691b9b58f8fffa55962b5dcd25/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/8350de41cfea3afa3960dd457147a795b9564fe18872330a8265484900b8efc5/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/1c72b4efdeca24b4fbe0317f123e2cd681a18d89a259c6f4c40bdcf905392454/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/5bcae3ee688f842ea070d6a083402c487b2ec0953bf7b904a4a4e91b80a74a99/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/52094bb875232746b90d005864b4650f93a86060fca098053d45bd00e29b5af8/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/39c90ed35f54543715b548bbc052aed2ac1268bf56a994de72a178f258d5e4b2/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/5ff563032ad3d0727076f523d557f80af76f1cc12e48815d11d2d466ff5ee548/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/2333621f61e84d547b5123be1acba685eb57ca39cea1912f9ccccf540c3f0f0c/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/d716c9a4ecd950d32fb1f7484ce8bd79323162fdf2a3c88ba380342bf405656d/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/bd8875042fd8e79b7837e0a61525bd484f4e8e7a7bed400d476ab196514b6c99/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/55492bca406fbe585485b4d15234056ef995f989c63b54a11efc23c915e6ac19/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f271dbec79b72360195a6cfcafde5d711ff051e63066701706f467c2c7a30b11/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3a98af49f5c58c7ecac5e1ba4f520e8df24c29465178f66ac3c89ef72d86abb7/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/d34276a8e4967444791d250e3a55407a6eada500ee71d5764a8152756a72ac32/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/6c1e177551134585d553464d1c4e26ce6b782748e1b771369f8ac177dee68f1a/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/dfc08a156f6d5556425af110d5dc53227f7ded733b1c915374786ca0abbe5249/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/f24c3b91a34b9e6d1710f6058363759d88b05aea955874d64794da9888eb5c94/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f329ad84a7472c76baf8bd5c3070b2bc239b6f62221685374b4514d31b8ec418/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/350610ea191bb680ec8e35f7d6f9936ed9f61feea753ae656291e4167213334a/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.463055 4968 manager.go:217] Machine: {Timestamp:2026-10-30 14:06:44.462558466 +0000 UTC m=+0.088001834 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105833 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21ff87d8ab27c55902169e3c643934 SystemUUID:ec21ff87-d8ab-27c5-5902-169e3c643934 BootID:8073ca32-e3f7-4345-aeb7-f19e25acf6c2 Filesystems:[{Device:/run/containers/storage/overlay-containers/bd6bd9f6c7a2ec2513e595e95e160b238a68d219ae90dcf25d7b31dc58fe390e/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e3af484b38bfa7e08e90b016ba031ea62141d0b5e580f3e77d688b42155a0d1/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1cdffa3f-7378-4f50-b1bc-e87aa647c4d8/volumes/kubernetes.io~projected/kube-api-access-tbcd9 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5e718a5497402d58bac6780c3a2e6c291eede489a5938ded3087c0e77797ddd/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e7a18c0a9391981fba99e22e260ba8b224ed58dc92119c0753c231079b552ca/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ead802d8-7ee2-4381-ad65-723736aabd12/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4171ff7a-5894-49d1-8683-606723fccdf9/volumes/kubernetes.io~projected/kube-api-access-ssmtf DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9ce8bbfd5ed5fcbbf19934621bfc94b2d35d455fb7ebf77f4b4c86bb21ae612/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66b11a57-8746-4a93-b726-a877c2a22a98/volumes/kubernetes.io~projected/kube-api-access-l2zxn DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe78a7b6-cba1-4c52-b190-0febeaab5035/volumes/kubernetes.io~projected/kube-api-access-lztlh DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aea273bc-03a8-41ad-8ed0-c1eab9a23a45/volumes/kubernetes.io~projected/kube-api-access-f7vhn DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/534e25f2af392926f0949ee3bb6b3799603647967a4fc967fe0e01ae8014d0dc/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/911b39fc-f2e6-4094-9be0-68606d2aa4f1/volumes/kubernetes.io~projected/kube-api-access-5vklh DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dfc6f25a-6a7a-472c-af78-683613ffb7a5/volumes/kubernetes.io~projected/kube-api-access-77kg8 DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed8f58363bcee5fbb7ff703de95bf9882033f5112ca4510aa71c383aa4acbd6f/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42511697-bcd4-4df3-819c-ab38ba5159c3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4d69a5c88e477db630b182c8faf58c2aa21014498416fa1d31968f939c475fd/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4070c760-6728-4166-a333-77af48851fd8/volumes/kubernetes.io~projected/kube-api-access-thjb5 DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c77829b61a63e7b3ca4e30adf669c8548309e83fb294025b95b5f22116da7cf/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70ab068a-ed6b-4fae-8019-2e7c5873ab43/volumes/kubernetes.io~projected/kube-api-access-9pwr6 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/354da4d4-65ce-4f03-a9aa-f7840bf28711/volumes/kubernetes.io~projected/kube-api-access-bcd7x DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/911b39fc-f2e6-4094-9be0-68606d2aa4f1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb9d3ab7-8a7f-4ccd-a336-912b00332110/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5/volumes/kubernetes.io~projected/kube-api-access-56lth DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~projected/kube-api-access-jv27n DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a0b0e80-8291-4adf-83a3-7732f79e0f80/volumes/kubernetes.io~projected/kube-api-access-p6pbv DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bc892c923ac74be04bf036a21cf1b28787d5c72dba173b2e407434e9be65521/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dfc6f25a-6a7a-472c-af78-683613ffb7a5/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~projected/kube-api-access-hxk4p DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78841e01-5bf7-4348-9802-09a81adeb88a/volumes/kubernetes.io~projected/kube-api-access-xjm94 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3db897157965152baeddf9496ee5d0dc0e65ea61064629d8ae00d218ac98cb1/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b388277-9bf2-45be-b85d-2eafd1a1c543/volumes/kubernetes.io~projected/kube-api-access-5xhlw DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a8ec17f-ff03-4af2-8885-1707bf84ec85/volumes/kubernetes.io~projected/kube-api-access-8qdqj DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70ab068a-ed6b-4fae-8019-2e7c5873ab43/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f40c0d7ae2c4ae218631ecef0226219b409c88f84c1cc8682c6fa9d2519ba61/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65/volumes/kubernetes.io~projected/kube-api-access-7sqjv DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/03049b65-310d-4ad7-b614-1daf3a50b528/volumes/kubernetes.io~projected/kube-api-access-f62nk DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4171ff7a-5894-49d1-8683-606723fccdf9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1804ec947805fc1806e9314ec207a3ffd212b9a4bc457bbd48a2ca0b28d02c2a/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4079ca19c89bdeaaf49de19f510759b96f2c4ba34a2b40ea6fcbe96b86afc2fd/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cc6f99f8e3efe6cf211c29d73d271b4b53c9b3d5936ef0937345119ff3b05d6/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3829121e-3790-47ff-b827-8774a70fec04/volumes/kubernetes.io~projected/kube-api-access-7w7rf DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66b11a57-8746-4a93-b726-a877c2a22a98/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4dca8d9b5a5f58da6d765ffd9051266f97f624e74a03bf7ec76b713465e78e2/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10c25997c409f0b29d95aa9f4484707384b72e3296304e35c7599b29249c9a5d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f07e86b-0018-4c02-a9c6-cc24bfecd068/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f07e86b-0018-4c02-a9c6-cc24bfecd068/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b612015fb112ba2dbc4456abd28335ab8c9f6ec986d7e3f0991c77b02c7502eb/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb9d3ab7-8a7f-4ccd-a336-912b00332110/volumes/kubernetes.io~projected/kube-api-access-928ww DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f07e86b-0018-4c02-a9c6-cc24bfecd068/volumes/kubernetes.io~projected/kube-api-access-wmq8x DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41/volumes/kubernetes.io~projected/kube-api-access-zw84x DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dfc6f25a-6a7a-472c-af78-683613ffb7a5/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6ae2497-ab63-4964-940a-e870d9131ae3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42511697-bcd4-4df3-819c-ab38ba5159c3/volumes/kubernetes.io~projected/kube-api-access-rrrj9 DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff67637528372b258ad2cc6dd8f09ed6aeedd40078fbacaf0659ec0c99e3f98e/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17aed83f-3fbe-4b2e-8714-e61122539496/volumes/kubernetes.io~projected/kube-api-access-82bmb DeviceMajor:0 DeviceMinor:239 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2b7943ae35026336881dcd417bae55e6e08df8ebcd6773537471075a3fa92d9/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70ab068a-ed6b-4fae-8019-2e7c5873ab43/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81e6ce71-7fb4-40ab-b14a-54457b97bd6a/volumes/kubernetes.io~projected/kube-api-access-cgn77 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75c66915c3d4f4e4d4c22877e594eb5c908adc64357a5dc45d1ff3dc30feff24/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4/volumes/kubernetes.io~projected/kube-api-access-9gjlh DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52aef7cad4bd812500483db121cfae723767e4715bcea840fb8cdb3305c03e9c/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51b981c69d8492be055c2dc1d8f7be2b52ce30979a988059582fc6a79a73b714/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e3b429d-b5ef-40bf-b31a-9bb2b5085010/volumes/kubernetes.io~projected/kube-api-access-gqwzb DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:4079ca19c89bdea MacAddress:36:98:b6:ac:b3:97 Speed:10000 Mtu:8901} {Name:4c77829b61a63e7 MacAddress:46:e2:4f:dd:cd:5b Speed:10000 Mtu:8901} {Name:4cc6f99f8e3efe6 MacAddress:32:cb:3b:ed:20:e9 Speed:10000 Mtu:8901} {Name:51b981c69d8492b MacAddress:b6:d4:5d:e5:7d:c0 Speed:10000 Mtu:8901} {Name:52aef7cad4bd812 MacAddress:92:c2:5c:69:b4:87 Speed:10000 Mtu:8901} {Name:5e7a18c0a939198 MacAddress:82:9e:bf:20:f5:43 Speed:10000 Mtu:8901} {Name:9e3af484b38bfa7 MacAddress:7a:0b:8d:88:77:f9 Speed:10000 Mtu:8901} {Name:a9ce8bbfd5ed5fc MacAddress:66:12:6b:e2:8b:93 Speed:10000 Mtu:8901} {Name:b5e718a5497402d MacAddress:fe:c4:3a:06:97:fd Speed:10000 Mtu:8901} {Name:b612015fb112ba2 MacAddress:fe:39:22:62:85:a0 Speed:10000 Mtu:8901} {Name:bd6bd9f6c7a2ec2 MacAddress:b2:26:42:0e:17:24 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d0:a1:ef:f9 Speed:0 Mtu:9001} {Name:br-int MacAddress:8e:0d:d7:da:bc:f7 Speed:0 Mtu:8901} {Name:c2b7943ae350263 MacAddress:82:ee:63:f0:38:46 Speed:10000 Mtu:8901} {Name:ed8f58363bcee5f MacAddress:4a:aa:b8:30:9e:d9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d0:a1:ef:f9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9a:75:f7:76:b0:14 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f2:b4:ff:3d:22:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.463403 4968 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.463446 4968 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.463733 4968 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.463759 4968 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-158.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.463892 4968 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.463900 4968 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.463910 4968 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.463923 4968 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.463942 4968 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.464117 4968 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.464224 4968 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.464258 4968 kubelet.go:493] "Attempting to sync node with API server" Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.464270 4968 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.464280 4968 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.464288 4968 kubelet.go:405] "Adding apiserver pod source" Oct 30 14:06:44.464314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.464296 4968 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 14:06:44.464986 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.464973 4968 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 14:06:44.464986 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.464985 4968 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:06:44.465449 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.465427 4968 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 14:06:44.467875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.467858 4968 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 14:06:44.467875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.467873 4968 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:06:44.467988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.467966 4968 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 14:06:44.468151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468140 4968 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 14:06:44.468186 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468169 4968 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 14:06:44.468263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468254 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 14:06:44.468292 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468268 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 14:06:44.468292 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468274 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 14:06:44.468292 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468279 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 14:06:44.468292 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468285 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 14:06:44.468292 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468291 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 14:06:44.468487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468297 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 14:06:44.468487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468303 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 14:06:44.468487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468310 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 14:06:44.468487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468316 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 14:06:44.468487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468324 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 14:06:44.468487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468335 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 14:06:44.468487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468358 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 14:06:44.468487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.468363 4968 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 14:06:44.471936 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.471923 4968 server.go:1263] "Started kubelet" Oct 30 14:06:44.471998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.471978 4968 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 14:06:44.472449 ip-10-0-129-158 systemd[1]: Started Kubernetes Kubelet. Oct 30 14:06:44.473297 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.473157 4968 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 14:06:44.473297 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.473263 4968 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 14:06:44.474084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.474061 4968 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 14:06:44.476577 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.475997 4968 server.go:317] "Adding debug handlers to kubelet server" Oct 30 14:06:44.478237 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.478221 4968 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 14:06:44.478341 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.478254 4968 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 14:06:44.478413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.478331 4968 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:01:05 +0000 UTC" deadline="2028-05-13 19:23:33.353401411 +0000 UTC" Oct 30 14:06:44.478413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.478353 4968 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13469h16m48.875051712s" Oct 30 14:06:44.478413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.478382 4968 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 14:06:44.478413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.478390 4968 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 14:06:44.478592 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:44.478461 4968 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found" Oct 30 14:06:44.478592 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.478478 4968 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 14:06:44.480677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480650 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4171ff7a-5894-49d1-8683-606723fccdf9" volumeName="kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-snapshots" seLinuxMountContext="" Oct 30 14:06:44.480812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480680 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f07e86b-0018-4c02-a9c6-cc24bfecd068" volumeName="kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-tmp" seLinuxMountContext="" Oct 30 14:06:44.480812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480697 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70ab068a-ed6b-4fae-8019-2e7c5873ab43" volumeName="kubernetes.io/secret/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 14:06:44.480812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480709 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aea273bc-03a8-41ad-8ed0-c1eab9a23a45" volumeName="kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn" seLinuxMountContext="" Oct 30 14:06:44.480812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480746 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfc6f25a-6a7a-472c-af78-683613ffb7a5" volumeName="kubernetes.io/projected/dfc6f25a-6a7a-472c-af78-683613ffb7a5-kube-api-access-77kg8" seLinuxMountContext="" Oct 30 14:06:44.480812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480757 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f753bb3b-c84d-4bd4-8b74-8ec50bb93c41" volumeName="kubernetes.io/projected/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kube-api-access-zw84x" seLinuxMountContext="" Oct 30 14:06:44.480812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480769 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb9d3ab7-8a7f-4ccd-a336-912b00332110" volumeName="kubernetes.io/configmap/fb9d3ab7-8a7f-4ccd-a336-912b00332110-config" seLinuxMountContext="" Oct 30 14:06:44.480812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480788 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb9d3ab7-8a7f-4ccd-a336-912b00332110" volumeName="kubernetes.io/projected/fb9d3ab7-8a7f-4ccd-a336-912b00332110-kube-api-access-928ww" seLinuxMountContext="" Oct 30 14:06:44.480812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480802 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5" volumeName="kubernetes.io/projected/a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5-kube-api-access-56lth" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480814 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42511697-bcd4-4df3-819c-ab38ba5159c3" volumeName="kubernetes.io/projected/42511697-bcd4-4df3-819c-ab38ba5159c3-kube-api-access-rrrj9" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480828 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65" volumeName="kubernetes.io/projected/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-kube-api-access-7sqjv" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480840 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4171ff7a-5894-49d1-8683-606723fccdf9" volumeName="kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-tmp" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480852 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b11a57-8746-4a93-b726-a877c2a22a98" volumeName="kubernetes.io/secret/66b11a57-8746-4a93-b726-a877c2a22a98-serving-cert" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480864 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78841e01-5bf7-4348-9802-09a81adeb88a" volumeName="kubernetes.io/projected/78841e01-5bf7-4348-9802-09a81adeb88a-kube-api-access-xjm94" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480879 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" volumeName="kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-binary-copy" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480892 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14b5c29-8cac-42f8-a3b2-6bf636cc05a4" volumeName="kubernetes.io/projected/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-kube-api-access-9gjlh" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480917 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a0b0e80-8291-4adf-83a3-7732f79e0f80" volumeName="kubernetes.io/projected/2a0b0e80-8291-4adf-83a3-7732f79e0f80-kube-api-access-p6pbv" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480930 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4171ff7a-5894-49d1-8683-606723fccdf9" volumeName="kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-trusted-ca-bundle" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480942 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" volumeName="kubernetes.io/projected/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-kube-api-access-cgn77" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480954 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd9fc616-08fa-480b-bd77-622e4dfbf59e" volumeName="kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-kube-api-access-hxk4p" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480965 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfc6f25a-6a7a-472c-af78-683613ffb7a5" volumeName="kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-default-certificate" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480977 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ead802d8-7ee2-4381-ad65-723736aabd12" volumeName="kubernetes.io/configmap/ead802d8-7ee2-4381-ad65-723736aabd12-konnectivity-ca" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.480988 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe78a7b6-cba1-4c52-b190-0febeaab5035" volumeName="kubernetes.io/projected/fe78a7b6-cba1-4c52-b190-0febeaab5035-kube-api-access-lztlh" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481002 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a8ec17f-ff03-4af2-8885-1707bf84ec85" volumeName="kubernetes.io/configmap/3a8ec17f-ff03-4af2-8885-1707bf84ec85-iptables-alerter-script" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481014 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e3b429d-b5ef-40bf-b31a-9bb2b5085010" volumeName="kubernetes.io/empty-dir/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-data-volume" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481027 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78841e01-5bf7-4348-9802-09a81adeb88a" volumeName="kubernetes.io/empty-dir/78841e01-5bf7-4348-9802-09a81adeb88a-tmp-dir" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481039 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" volumeName="kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481051 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd07a434-a5f6-4cda-a93d-8824978aa590" volumeName="kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-installation-pull-secrets" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481069 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4171ff7a-5894-49d1-8683-606723fccdf9" volumeName="kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-service-ca-bundle" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481081 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b388277-9bf2-45be-b85d-2eafd1a1c543" volumeName="kubernetes.io/projected/5b388277-9bf2-45be-b85d-2eafd1a1c543-kube-api-access-5xhlw" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481093 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b11a57-8746-4a93-b726-a877c2a22a98" volumeName="kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-trusted-ca" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481105 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70ab068a-ed6b-4fae-8019-2e7c5873ab43" volumeName="kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-script-lib" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481117 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd9fc616-08fa-480b-bd77-622e4dfbf59e" volumeName="kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-bound-sa-token" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481128 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42511697-bcd4-4df3-819c-ab38ba5159c3" volumeName="kubernetes.io/configmap/42511697-bcd4-4df3-819c-ab38ba5159c3-config" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481140 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b11a57-8746-4a93-b726-a877c2a22a98" volumeName="kubernetes.io/projected/66b11a57-8746-4a93-b726-a877c2a22a98-kube-api-access-l2zxn" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481152 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" volumeName="kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481163 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="911b39fc-f2e6-4094-9be0-68606d2aa4f1" volumeName="kubernetes.io/secret/911b39fc-f2e6-4094-9be0-68606d2aa4f1-cert" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481175 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ead802d8-7ee2-4381-ad65-723736aabd12" volumeName="kubernetes.io/secret/ead802d8-7ee2-4381-ad65-723736aabd12-agent-certs" seLinuxMountContext="" Oct 30 14:06:44.481179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481186 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42511697-bcd4-4df3-819c-ab38ba5159c3" volumeName="kubernetes.io/secret/42511697-bcd4-4df3-819c-ab38ba5159c3-serving-cert" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481198 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd9fc616-08fa-480b-bd77-622e4dfbf59e" volumeName="kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-image-registry-private-configuration" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481211 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6ae2497-ab63-4964-940a-e870d9131ae3" volumeName="kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481223 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd07a434-a5f6-4cda-a93d-8824978aa590" volumeName="kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-trusted-ca" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481236 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd07a434-a5f6-4cda-a93d-8824978aa590" volumeName="kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-certificates" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481247 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17aed83f-3fbe-4b2e-8714-e61122539496" volumeName="kubernetes.io/projected/17aed83f-3fbe-4b2e-8714-e61122539496-kube-api-access-82bmb" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481260 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b388277-9bf2-45be-b85d-2eafd1a1c543" volumeName="kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-daemon-config" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481272 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70ab068a-ed6b-4fae-8019-2e7c5873ab43" volumeName="kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-env-overrides" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481284 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb9d3ab7-8a7f-4ccd-a336-912b00332110" volumeName="kubernetes.io/secret/fb9d3ab7-8a7f-4ccd-a336-912b00332110-serving-cert" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481295 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd9fc616-08fa-480b-bd77-622e4dfbf59e" volumeName="kubernetes.io/empty-dir/bd9fc616-08fa-480b-bd77-622e4dfbf59e-ca-trust-extracted" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481307 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f07e86b-0018-4c02-a9c6-cc24bfecd068" volumeName="kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-tuned" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481318 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e3b429d-b5ef-40bf-b31a-9bb2b5085010" volumeName="kubernetes.io/projected/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-api-access-gqwzb" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481330 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3829121e-3790-47ff-b827-8774a70fec04" volumeName="kubernetes.io/projected/3829121e-3790-47ff-b827-8774a70fec04-kube-api-access-7w7rf" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481341 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4171ff7a-5894-49d1-8683-606723fccdf9" volumeName="kubernetes.io/projected/4171ff7a-5894-49d1-8683-606723fccdf9-kube-api-access-ssmtf" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481354 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd07a434-a5f6-4cda-a93d-8824978aa590" volumeName="kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-kube-api-access-jv27n" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481366 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17aed83f-3fbe-4b2e-8714-e61122539496" volumeName="kubernetes.io/configmap/17aed83f-3fbe-4b2e-8714-e61122539496-config-volume" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481378 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17aed83f-3fbe-4b2e-8714-e61122539496" volumeName="kubernetes.io/empty-dir/17aed83f-3fbe-4b2e-8714-e61122539496-tmp-dir" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481392 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="354da4d4-65ce-4f03-a9aa-f7840bf28711" volumeName="kubernetes.io/configmap/354da4d4-65ce-4f03-a9aa-f7840bf28711-serviceca" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481418 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b388277-9bf2-45be-b85d-2eafd1a1c543" volumeName="kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-cni-binary-copy" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481430 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd07a434-a5f6-4cda-a93d-8824978aa590" volumeName="kubernetes.io/empty-dir/fd07a434-a5f6-4cda-a93d-8824978aa590-ca-trust-extracted" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481443 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4070c760-6728-4166-a333-77af48851fd8" volumeName="kubernetes.io/projected/4070c760-6728-4166-a333-77af48851fd8-kube-api-access-thjb5" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481454 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4171ff7a-5894-49d1-8683-606723fccdf9" volumeName="kubernetes.io/secret/4171ff7a-5894-49d1-8683-606723fccdf9-serving-cert" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481466 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f07e86b-0018-4c02-a9c6-cc24bfecd068" volumeName="kubernetes.io/projected/4f07e86b-0018-4c02-a9c6-cc24bfecd068-kube-api-access-wmq8x" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481482 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b11a57-8746-4a93-b726-a877c2a22a98" volumeName="kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-config" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481493 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd9fc616-08fa-480b-bd77-622e4dfbf59e" volumeName="kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-certificates" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481509 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cdffa3f-7378-4f50-b1bc-e87aa647c4d8" volumeName="kubernetes.io/projected/1cdffa3f-7378-4f50-b1bc-e87aa647c4d8-kube-api-access-tbcd9" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481522 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="911b39fc-f2e6-4094-9be0-68606d2aa4f1" volumeName="kubernetes.io/projected/911b39fc-f2e6-4094-9be0-68606d2aa4f1-kube-api-access-5vklh" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481533 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03049b65-310d-4ad7-b614-1daf3a50b528" volumeName="kubernetes.io/projected/03049b65-310d-4ad7-b614-1daf3a50b528-kube-api-access-f62nk" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481545 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4070c760-6728-4166-a333-77af48851fd8" volumeName="kubernetes.io/configmap/4070c760-6728-4166-a333-77af48851fd8-samples-operator-config" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481560 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70ab068a-ed6b-4fae-8019-2e7c5873ab43" volumeName="kubernetes.io/projected/70ab068a-ed6b-4fae-8019-2e7c5873ab43-kube-api-access-9pwr6" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481573 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="354da4d4-65ce-4f03-a9aa-f7840bf28711" volumeName="kubernetes.io/projected/354da4d4-65ce-4f03-a9aa-f7840bf28711-kube-api-access-bcd7x" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481584 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a8ec17f-ff03-4af2-8885-1707bf84ec85" volumeName="kubernetes.io/projected/3a8ec17f-ff03-4af2-8885-1707bf84ec85-kube-api-access-8qdqj" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481600 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70ab068a-ed6b-4fae-8019-2e7c5873ab43" volumeName="kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-config" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481612 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd9fc616-08fa-480b-bd77-622e4dfbf59e" volumeName="kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-trusted-ca" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481624 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd9fc616-08fa-480b-bd77-622e4dfbf59e" volumeName="kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-installation-pull-secrets" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481639 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfc6f25a-6a7a-472c-af78-683613ffb7a5" volumeName="kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-stats-auth" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481651 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65" volumeName="kubernetes.io/configmap/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-telemetry-config" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481662 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2536dd8-0f2c-4571-b0db-484894b2a99f" volumeName="kubernetes.io/configmap/c2536dd8-0f2c-4571-b0db-484894b2a99f-nginx-conf" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481704 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd07a434-a5f6-4cda-a93d-8824978aa590" volumeName="kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-bound-sa-token" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481737 4968 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd07a434-a5f6-4cda-a93d-8824978aa590" volumeName="kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-image-registry-private-configuration" seLinuxMountContext="" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481750 4968 reconstruct.go:97] "Volume reconstruction finished" Oct 30 14:06:44.482587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.481756 4968 reconciler.go:29] "Reconciler: start to sync state" Oct 30 14:06:44.486380 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.486344 4968 factory.go:55] Registering systemd factory Oct 30 14:06:44.486380 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.486364 4968 factory.go:223] Registration of the systemd container factory successfully Oct 30 14:06:44.486929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.486589 4968 factory.go:153] Registering CRI-O factory Oct 30 14:06:44.486929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.486605 4968 factory.go:223] Registration of the crio container factory successfully Oct 30 14:06:44.486929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.486655 4968 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 14:06:44.486929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.486677 4968 factory.go:103] Registering Raw factory Oct 30 14:06:44.486929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.486689 4968 manager.go:1199] Started watching for new ooms in manager Oct 30 14:06:44.487170 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.487053 4968 manager.go:319] Starting recovery of all containers Oct 30 14:06:44.495817 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.495788 4968 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 14:06:44.497982 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.497954 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/1.log" Oct 30 14:06:44.498527 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.498511 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/1.log" Oct 30 14:06:44.510767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.510743 4968 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 14:06:44.510767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.510769 4968 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 14:06:44.510930 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.510791 4968 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 14:06:44.510930 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:44.510837 4968 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 14:06:44.523866 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.523840 4968 generic.go:356] "Generic (PLEG): container finished" podID="51d3226536491d82b487daab954fd946" containerID="f641e47ff4b7e70294a2c16788c914c68916a1a4e0f000a35969e948c14fe276" exitCode=0 Oct 30 14:06:44.528160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.528136 4968 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:06:44.528374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.528353 4968 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 14:06:44.529624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.529606 4968 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:06:44.532747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.532708 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/1.log" Oct 30 14:06:44.532833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.532755 4968 generic.go:356] "Generic (PLEG): container finished" podID="66b11a57-8746-4a93-b726-a877c2a22a98" containerID="6f398efb0490e4b545d0c2f056c5615256be15aa38b06eb15dad2a77ff08d592" exitCode=255 Oct 30 14:06:44.539540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.539522 4968 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:06:44.539753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.539732 4968 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:06:44.560657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.560634 4968 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="779fac208966284b3d37094680e967633749bd27edc478c47043643dfbbaeb50" exitCode=0 Oct 30 14:06:44.560657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.560656 4968 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="3c46a6f54ad64892eeae35d566190b1b6e2a39343d551fc01bb1ca11fece5f02" exitCode=0 Oct 30 14:06:44.560828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.560665 4968 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="0791491ef5384b4d7ea36686a4d18725b019c51e560c5007b48b4236f71d6991" exitCode=0 Oct 30 14:06:44.560828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.560672 4968 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="88f45a489cd1e0d10ccd3ce1019295c3cc367ac15e69f2b88cbb753e60e52cf8" exitCode=0 Oct 30 14:06:44.560828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.560677 4968 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="26cb673bf2e93b36ad2a67f2bcfc95c778a59a25755c5273c6721f5a54832654" exitCode=0 Oct 30 14:06:44.560828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.560682 4968 generic.go:356] "Generic (PLEG): container finished" podID="81e6ce71-7fb4-40ab-b14a-54457b97bd6a" containerID="858e634874a0b293aea01d173c794374317a2cdaaa7ff62072a6766b551b7723" exitCode=0 Oct 30 14:06:44.577680 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.577662 4968 manager.go:324] Recovery completed Oct 30 14:06:44.598609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.598586 4968 cpu_manager.go:226] "Starting" policy="none" Oct 30 14:06:44.598609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.598605 4968 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 14:06:44.598778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.598624 4968 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 14:06:44.598778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.598757 4968 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 14:06:44.598860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.598773 4968 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 14:06:44.598860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.598790 4968 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 14:06:44.598860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.598795 4968 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 14:06:44.598860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.598801 4968 policy_none.go:50] "Start" Oct 30 14:06:44.598860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.598837 4968 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 14:06:44.598860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.598851 4968 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 14:06:44.599010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.598999 4968 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 14:06:44.599038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.599009 4968 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 14:06:44.599038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.599017 4968 policy_none.go:44] "Start" Oct 30 14:06:44.603348 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.603325 4968 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 14:06:44.605911 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.605896 4968 manager.go:341] "Starting Device Plugin manager" Oct 30 14:06:44.606354 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:44.605927 4968 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 14:06:44.606434 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.606358 4968 server.go:87] "Starting device plugin registration server" Oct 30 14:06:44.606661 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.606648 4968 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 14:06:44.606746 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.606662 4968 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 14:06:44.606798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.606757 4968 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 14:06:44.606891 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.606854 4968 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 14:06:44.606891 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.606894 4968 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 14:06:44.607051 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.606901 4968 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 14:06:44.607051 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.606989 4968 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T14:06:44.606884578Z","UUID":"f53ab4a4-5d0a-4264-b4f0-83cd20a94032","Handler":null,"Name":"","Endpoint":""} Oct 30 14:06:44.608306 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.608287 4968 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 14:06:44.608393 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.608313 4968 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 14:06:44.608513 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:44.608492 4968 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 14:06:44.610909 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.610874 4968 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal"] Oct 30 14:06:44.611516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.611490 4968 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.611628 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.611597 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" event={"ID":"274ce57e4715bf968542a8e94100b125","Type":"ContainerStarted","Data":"029b3058822bef799ba07838fbefb92bcc78629ee7ebbc0a6821db25bb93dc5f"} Oct 30 14:06:44.611672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.611640 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" event={"ID":"274ce57e4715bf968542a8e94100b125","Type":"ContainerStarted","Data":"f4dca8d9b5a5f58da6d765ffd9051266f97f624e74a03bf7ec76b713465e78e2"} Oct 30 14:06:44.611672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.611660 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" event={"ID":"51d3226536491d82b487daab954fd946","Type":"ContainerStarted","Data":"09bf5ad456fba4d82631471d20fec0fe03ffb0a449291b3c7ba6d566d1535435"} Oct 30 14:06:44.611747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.611678 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" event={"ID":"51d3226536491d82b487daab954fd946","Type":"ContainerDied","Data":"f641e47ff4b7e70294a2c16788c914c68916a1a4e0f000a35969e948c14fe276"} Oct 30 14:06:44.611747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.611697 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" event={"ID":"51d3226536491d82b487daab954fd946","Type":"ContainerStarted","Data":"0bc892c923ac74be04bf036a21cf1b28787d5c72dba173b2e407434e9be65521"} Oct 30 14:06:44.611882 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.611863 4968 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.636598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.636579 4968 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:06:44.636681 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:44.636615 4968 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.637557 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.637518 4968 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:06:44.639453 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:44.639423 4968 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-158.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.707601 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.707543 4968 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:06:44.708767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.708749 4968 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:06:44.708829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.708779 4968 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:06:44.708829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.708789 4968 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:06:44.708829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.708826 4968 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.728729 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.728692 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.728807 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.728737 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.728807 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.728753 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/274ce57e4715bf968542a8e94100b125-config\") pod \"kube-apiserver-proxy-ip-10-0-129-158.ec2.internal\" (UID: \"274ce57e4715bf968542a8e94100b125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.741010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.740991 4968 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.741067 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.741051 4968 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.829143 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.829113 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.829143 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.829146 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.829295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.829163 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/274ce57e4715bf968542a8e94100b125-config\") pod \"kube-apiserver-proxy-ip-10-0-129-158.ec2.internal\" (UID: \"274ce57e4715bf968542a8e94100b125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.829295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.829217 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.829295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.829227 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51d3226536491d82b487daab954fd946-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"51d3226536491d82b487daab954fd946\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:44.829295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:44.829266 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/274ce57e4715bf968542a8e94100b125-config\") pod \"kube-apiserver-proxy-ip-10-0-129-158.ec2.internal\" (UID: \"274ce57e4715bf968542a8e94100b125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:45.465028 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.464993 4968 apiserver.go:52] "Watching apiserver" Oct 30 14:06:45.508268 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.508238 4968 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 14:06:45.508919 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.508898 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal","openshift-ingress-canary/ingress-canary-m8jt7","openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm","openshift-multus/multus-fp57b","openshift-insights/insights-operator-7c4986d449-tfvs9","openshift-image-registry/image-registry-8fff9cc46-btf9d","openshift-console-operator/console-operator-7cd56b8dbc-84mcb","openshift-ovn-kubernetes/ovnkube-node-clqhk","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh","openshift-ingress/router-default-598ddffcff-nctk8","kube-system/kas-connection-checker-685895c7c9-f298s","kube-system/konnectivity-agent-h6hnr","openshift-dns/node-resolver-qnfg6","openshift-image-registry/image-registry-89594b78c-qxvnh","openshift-image-registry/node-ca-vpkj5","kube-system/kas-connection-checker-685895c7c9-dmj28","openshift-network-operator/iptables-alerter-xf6z2","openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p","openshift-multus/network-metrics-daemon-rgjxr","kube-system/global-pull-secret-syncer-vlmdd","openshift-cluster-node-tuning-operator/tuned-crmmp","openshift-insights/insights-runtime-extractor-hmtlt","openshift-network-diagnostics/network-check-target-stdbh","openshift-dns/dns-default-fvkvc","openshift-network-console/networking-console-plugin-86c478bb88-w9s2h","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb","openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz","kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4","kube-system/kas-connection-checker-685895c7c9-jmrnq","openshift-multus/multus-additional-cni-plugins-mssbd"] Oct 30 14:06:45.510271 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.510250 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:45.511031 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.511007 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.512296 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.512278 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:45.513068 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.513054 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:45.513797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.513781 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:45.516603 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.516586 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:45.522657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.522638 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.522760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.522693 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.522760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.522642 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.522913 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.522896 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 14:06:45.522970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.522950 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.523018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.522962 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 14:06:45.523018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.522975 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.523018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.523003 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.523233 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.523215 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.523335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.523241 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:45.523826 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.523803 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 14:06:45.523826 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.523821 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 14:06:45.524054 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.523847 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 14:06:45.524054 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.523849 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 14:06:45.524054 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.523909 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 14:06:45.524054 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.523915 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 14:06:45.524289 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.524269 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.524337 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.524311 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 14:06:45.524502 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.524489 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.524614 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.524597 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 14:06:45.531226 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.531211 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 14:06:45.531487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.531471 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 14:06:45.531487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.531480 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 14:06:45.531735 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.531709 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 14:06:45.533509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.533490 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 14:06:45.533650 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.533572 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9wxhp\"" Oct 30 14:06:45.533853 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.533800 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 14:06:45.533853 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.533845 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.533980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.533856 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.533980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.533873 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w5lxm\"" Oct 30 14:06:45.533980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.533958 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 14:06:45.534178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.533985 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 14:06:45.534178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534004 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 14:06:45.534178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534011 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.534178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534019 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.534178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534014 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.534178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534002 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.534178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534042 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 14:06:45.534178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534125 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534243 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534259 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534264 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534261 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534280 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534304 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534313 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534325 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534328 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534375 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534521 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.534538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534546 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534548 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534558 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534567 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534571 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534593 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534601 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534694 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534763 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534766 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qkz6n\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534772 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.534773 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 14:06:45.535071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535068 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.535570 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535309 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mlbs5\"" Oct 30 14:06:45.535570 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535327 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.535570 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535366 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.535570 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535371 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 14:06:45.535570 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535380 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 14:06:45.535570 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535373 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.535570 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535445 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mfmhk\"" Oct 30 14:06:45.535570 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535565 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 14:06:45.535998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535593 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 14:06:45.535998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535604 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c5nrs\"" Oct 30 14:06:45.535998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535740 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2dqn6\"" Oct 30 14:06:45.535998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.535788 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgz8z\"" Oct 30 14:06:45.536119 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.536092 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 14:06:45.536368 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.536355 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 14:06:45.536469 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.536367 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.536604 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.536588 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5kvlg\"" Oct 30 14:06:45.536604 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.536591 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7x2qs\"" Oct 30 14:06:45.536684 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.536599 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 14:06:45.536684 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.536643 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5t6gk\"" Oct 30 14:06:45.544782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.544760 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-94q7c\"" Oct 30 14:06:45.544874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.544770 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 14:06:45.544960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.544947 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 14:06:45.545018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.544947 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dqvfb\"" Oct 30 14:06:45.545067 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545015 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.545115 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545094 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.545163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545123 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 14:06:45.545163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545150 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.545305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545286 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 14:06:45.545358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545302 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.545468 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545445 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4szpv\"" Oct 30 14:06:45.545571 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545555 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4w84p\"" Oct 30 14:06:45.545772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545754 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 14:06:45.545867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545786 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 14:06:45.545867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545811 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g7grg\"" Oct 30 14:06:45.546011 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545997 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 14:06:45.546077 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.546005 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z4lqh\"" Oct 30 14:06:45.546077 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.545997 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vfwlz\"" Oct 30 14:06:45.546172 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.546116 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 14:06:45.546172 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.546159 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 14:06:45.547143 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.546561 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 14:06:45.566644 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.566622 4968 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:45.566806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.566785 4968 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:45.580434 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.580406 4968 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 14:06:45.585973 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.585957 4968 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:06:45.586029 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.586000 4968 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-158.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" Oct 30 14:06:45.586831 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.586819 4968 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:06:45.586881 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.586849 4968 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" Oct 30 14:06:45.632633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.632605 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-sys-fs\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.632633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.632643 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw84x\" (UniqueName: \"kubernetes.io/projected/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kube-api-access-zw84x\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.632868 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.632671 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd9fc616-08fa-480b-bd77-622e4dfbf59e-ca-trust-extracted\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.632914 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.632871 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd9fc616-08fa-480b-bd77-622e4dfbf59e-ca-trust-extracted\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.632949 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.632917 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-netns\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.632949 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.632938 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-k8s-cni-cncf-io\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.633033 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.632954 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-kubelet\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.633033 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.632973 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-conf-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.633033 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.632994 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd07a434-a5f6-4cda-a93d-8824978aa590-ca-trust-extracted\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.633144 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633045 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd07a434-a5f6-4cda-a93d-8824978aa590-ca-trust-extracted\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.633144 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633044 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.633144 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633081 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a8ec17f-ff03-4af2-8885-1707bf84ec85-iptables-alerter-script\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:45.633144 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633101 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:45.633298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633150 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysconfig\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.633298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633168 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-device-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.633298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633189 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ead802d8-7ee2-4381-ad65-723736aabd12-agent-certs\") pod \"konnectivity-agent-h6hnr\" (UID: \"ead802d8-7ee2-4381-ad65-723736aabd12\") " pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:45.633298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633225 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-snapshots\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.633298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633275 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.633298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633292 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-snapshots\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.633540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633322 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.633540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633401 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a8ec17f-ff03-4af2-8885-1707bf84ec85-iptables-alerter-script\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:45.633540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633450 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6pbv\" (UniqueName: \"kubernetes.io/projected/2a0b0e80-8291-4adf-83a3-7732f79e0f80-kube-api-access-p6pbv\") pod \"migrator-66df5d496b-42qwv\" (UID: \"2a0b0e80-8291-4adf-83a3-7732f79e0f80\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv" Oct 30 14:06:45.633540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633492 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcd7x\" (UniqueName: \"kubernetes.io/projected/354da4d4-65ce-4f03-a9aa-f7840bf28711-kube-api-access-bcd7x\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:45.633540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633519 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqjv\" (UniqueName: \"kubernetes.io/projected/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-kube-api-access-7sqjv\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:45.633540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633524 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ead802d8-7ee2-4381-ad65-723736aabd12-agent-certs\") pod \"konnectivity-agent-h6hnr\" (UID: \"ead802d8-7ee2-4381-ad65-723736aabd12\") " pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:45.633874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633548 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-trusted-ca\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:45.633874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633573 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4171ff7a-5894-49d1-8683-606723fccdf9-serving-cert\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.633874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633597 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmtf\" (UniqueName: \"kubernetes.io/projected/4171ff7a-5894-49d1-8683-606723fccdf9-kube-api-access-ssmtf\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.633874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633662 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.633874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633825 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqwzb\" (UniqueName: \"kubernetes.io/projected/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-api-access-gqwzb\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.633874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633863 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-modprobe-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.634045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633881 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.634045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633828 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4171ff7a-5894-49d1-8683-606723fccdf9-serving-cert\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.634045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.633907 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmq8x\" (UniqueName: \"kubernetes.io/projected/4f07e86b-0018-4c02-a9c6-cc24bfecd068-kube-api-access-wmq8x\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.634130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634070 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-trusted-ca\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:45.634130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634083 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.634199 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634131 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:45.634228 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634203 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2536dd8-0f2c-4571-b0db-484894b2a99f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:45.634257 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634242 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-bin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.634285 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634264 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-run\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.634313 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634289 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2zxn\" (UniqueName: \"kubernetes.io/projected/66b11a57-8746-4a93-b726-a877c2a22a98-kube-api-access-l2zxn\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:45.634363 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634347 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-script-lib\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.634405 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634382 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-multus-certs\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.634452 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634413 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-bound-sa-token\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.634452 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634424 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.634452 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634438 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-log-socket\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.634452 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634443 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2536dd8-0f2c-4571-b0db-484894b2a99f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:45.634641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634551 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjm94\" (UniqueName: \"kubernetes.io/projected/78841e01-5bf7-4348-9802-09a81adeb88a-kube-api-access-xjm94\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:45.634641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634579 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-trusted-ca\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.634641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634605 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-multus\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.634641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634631 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-daemon-config\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.634900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634655 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-certificates\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.634900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634683 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-var-lib-kubelet\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.634900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634706 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17aed83f-3fbe-4b2e-8714-e61122539496-tmp-dir\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:45.634900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634704 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-script-lib\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.634900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634745 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:45.634900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634763 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxk4p\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-kube-api-access-hxk4p\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.634900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634780 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thjb5\" (UniqueName: \"kubernetes.io/projected/4070c760-6728-4166-a333-77af48851fd8-kube-api-access-thjb5\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:45.634900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634822 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17aed83f-3fbe-4b2e-8714-e61122539496-tmp-dir\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634929 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-daemon-config\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634984 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-trusted-ca\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635002 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.634989 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-stats-auth\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635047 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-certificates\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635048 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-image-registry-private-configuration\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635092 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-service-ca-bundle\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635130 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/911b39fc-f2e6-4094-9be0-68606d2aa4f1-cert\") pod \"ingress-canary-m8jt7\" (UID: \"911b39fc-f2e6-4094-9be0-68606d2aa4f1\") " pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635156 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-dbus\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635199 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-stats-auth\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635201 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-systemd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635231 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-image-registry-private-configuration\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.635307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635244 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-ovn\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635317 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4171ff7a-5894-49d1-8683-606723fccdf9-service-ca-bundle\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635367 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ead802d8-7ee2-4381-ad65-723736aabd12-konnectivity-ca\") pod \"konnectivity-agent-h6hnr\" (UID: \"ead802d8-7ee2-4381-ad65-723736aabd12\") " pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635389 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-tmp\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635395 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/911b39fc-f2e6-4094-9be0-68606d2aa4f1-cert\") pod \"ingress-canary-m8jt7\" (UID: \"911b39fc-f2e6-4094-9be0-68606d2aa4f1\") " pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635406 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-certificates\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635455 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4171ff7a-5894-49d1-8683-606723fccdf9-tmp\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635491 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9d3ab7-8a7f-4ccd-a336-912b00332110-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635498 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ead802d8-7ee2-4381-ad65-723736aabd12-konnectivity-ca\") pod \"konnectivity-agent-h6hnr\" (UID: \"ead802d8-7ee2-4381-ad65-723736aabd12\") " pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635508 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-etc-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635527 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-certificates\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635527 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-installation-pull-secrets\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635612 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-trusted-ca\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.635647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635627 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9d3ab7-8a7f-4ccd-a336-912b00332110-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635658 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-installation-pull-secrets\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635659 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-928ww\" (UniqueName: \"kubernetes.io/projected/fb9d3ab7-8a7f-4ccd-a336-912b00332110-kube-api-access-928ww\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635690 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztlh\" (UniqueName: \"kubernetes.io/projected/fe78a7b6-cba1-4c52-b190-0febeaab5035-kube-api-access-lztlh\") pod \"network-check-source-6f78d59f99-2sxp4\" (UID: \"fe78a7b6-cba1-4c52-b190-0febeaab5035\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635738 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635805 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovn-node-metrics-cert\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635861 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42511697-bcd4-4df3-819c-ab38ba5159c3-serving-cert\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635896 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv27n\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-kube-api-access-jv27n\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635922 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17aed83f-3fbe-4b2e-8714-e61122539496-config-volume\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635954 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-socket-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635983 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4070c760-6728-4166-a333-77af48851fd8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.635981 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-trusted-ca\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636007 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42511697-bcd4-4df3-819c-ab38ba5159c3-serving-cert\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636019 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-default-certificate\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636048 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-conf\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636070 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovn-node-metrics-cert\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636072 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66b11a57-8746-4a93-b726-a877c2a22a98-serving-cert\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636105 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636132 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-installation-pull-secrets\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636178 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636188 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17aed83f-3fbe-4b2e-8714-e61122539496-config-volume\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:45.636206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636194 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4070c760-6728-4166-a333-77af48851fd8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636238 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42511697-bcd4-4df3-819c-ab38ba5159c3-config\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636264 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-default-certificate\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636258 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636300 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636323 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-systemd\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636352 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bmb\" (UniqueName: \"kubernetes.io/projected/17aed83f-3fbe-4b2e-8714-e61122539496-kube-api-access-82bmb\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636379 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66b11a57-8746-4a93-b726-a877c2a22a98-serving-cert\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636385 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42511697-bcd4-4df3-819c-ab38ba5159c3-config\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636393 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636420 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-node-log\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636441 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-installation-pull-secrets\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636446 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-config\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636474 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-kubernetes\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636565 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-host\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636591 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb9d3ab7-8a7f-4ccd-a336-912b00332110-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636606 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-ovnkube-config\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636645 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636666 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-etc-kubernetes\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636685 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgn77\" (UniqueName: \"kubernetes.io/projected/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-kube-api-access-cgn77\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636734 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjlh\" (UniqueName: \"kubernetes.io/projected/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-kube-api-access-9gjlh\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636768 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb9d3ab7-8a7f-4ccd-a336-912b00332110-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636764 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-netd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636816 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pwr6\" (UniqueName: \"kubernetes.io/projected/70ab068a-ed6b-4fae-8019-2e7c5873ab43-kube-api-access-9pwr6\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636911 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-system-cni-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636949 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-bound-sa-token\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.637009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.636992 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-env-overrides\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637023 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cnibin\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637053 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbcd9\" (UniqueName: \"kubernetes.io/projected/1cdffa3f-7378-4f50-b1bc-e87aa647c4d8-kube-api-access-tbcd9\") pod \"kas-connection-checker-685895c7c9-jmrnq\" (UID: \"1cdffa3f-7378-4f50-b1bc-e87aa647c4d8\") " pod="kube-system/kas-connection-checker-685895c7c9-jmrnq" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637100 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-image-registry-private-configuration\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637153 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70ab068a-ed6b-4fae-8019-2e7c5873ab43-env-overrides\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637190 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-systemd-units\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637224 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-hostroot\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637256 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-image-registry-private-configuration\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637269 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62nk\" (UniqueName: \"kubernetes.io/projected/03049b65-310d-4ad7-b614-1daf3a50b528-kube-api-access-f62nk\") pod \"volume-data-source-validator-58c9f5f46d-nw947\" (UID: \"03049b65-310d-4ad7-b614-1daf3a50b528\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637403 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637432 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637451 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637480 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/354da4d4-65ce-4f03-a9aa-f7840bf28711-host\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637561 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637586 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhlw\" (UniqueName: \"kubernetes.io/projected/5b388277-9bf2-45be-b85d-2eafd1a1c543-kube-api-access-5xhlw\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637610 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6ae2497-ab63-4964-940a-e870d9131ae3-original-pull-secret\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637619 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-lib-modules\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637728 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637757 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-registration-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637790 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-kubelet-config\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637812 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-config\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637828 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a8ec17f-ff03-4af2-8885-1707bf84ec85-host-slash\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:45.637835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637844 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/354da4d4-65ce-4f03-a9aa-f7840bf28711-serviceca\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637943 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637979 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56lth\" (UniqueName: \"kubernetes.io/projected/a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5-kube-api-access-56lth\") pod \"kas-connection-checker-685895c7c9-f298s\" (UID: \"a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5\") " pod="kube-system/kas-connection-checker-685895c7c9-f298s" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.637991 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66b11a57-8746-4a93-b726-a877c2a22a98-config\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638013 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-system-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638041 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638055 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/354da4d4-65ce-4f03-a9aa-f7840bf28711-serviceca\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638067 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qdqj\" (UniqueName: \"kubernetes.io/projected/3a8ec17f-ff03-4af2-8885-1707bf84ec85-kube-api-access-8qdqj\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638092 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638116 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-socket-dir-parent\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638142 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78841e01-5bf7-4348-9802-09a81adeb88a-tmp-dir\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638219 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78841e01-5bf7-4348-9802-09a81adeb88a-tmp-dir\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638234 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638296 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77kg8\" (UniqueName: \"kubernetes.io/projected/dfc6f25a-6a7a-472c-af78-683613ffb7a5-kube-api-access-77kg8\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638338 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrj9\" (UniqueName: \"kubernetes.io/projected/42511697-bcd4-4df3-819c-ab38ba5159c3-kube-api-access-rrrj9\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638356 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638378 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-var-lib-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638394 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638414 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-netns\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638454 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-tuned\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638486 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7rf\" (UniqueName: \"kubernetes.io/projected/3829121e-3790-47ff-b827-8774a70fec04-kube-api-access-7w7rf\") pod \"kas-connection-checker-685895c7c9-dmj28\" (UID: \"3829121e-3790-47ff-b827-8774a70fec04\") " pod="kube-system/kas-connection-checker-685895c7c9-dmj28" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638536 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-slash\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638563 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-bin\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638597 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-cnibin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.638624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638599 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-tuned\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638638 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78841e01-5bf7-4348-9802-09a81adeb88a-hosts-file\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638681 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638702 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vhn\" (UniqueName: \"kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn\") pod \"network-check-target-stdbh\" (UID: \"aea273bc-03a8-41ad-8ed0-c1eab9a23a45\") " pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638745 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-sys\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638764 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-tmp\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638782 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-kubelet\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638797 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-crio-socket\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638815 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-data-volume\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638864 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-data-volume\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638869 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f07e86b-0018-4c02-a9c6-cc24bfecd068-tmp\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638902 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-cni-binary-copy\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.638934 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vklh\" (UniqueName: \"kubernetes.io/projected/911b39fc-f2e6-4094-9be0-68606d2aa4f1-kube-api-access-5vklh\") pod \"ingress-canary-m8jt7\" (UID: \"911b39fc-f2e6-4094-9be0-68606d2aa4f1\") " pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:45.639381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.639078 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b388277-9bf2-45be-b85d-2eafd1a1c543-cni-binary-copy\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.653772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.653752 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcd7x\" (UniqueName: \"kubernetes.io/projected/354da4d4-65ce-4f03-a9aa-f7840bf28711-kube-api-access-bcd7x\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:45.662556 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.662535 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6pbv\" (UniqueName: \"kubernetes.io/projected/2a0b0e80-8291-4adf-83a3-7732f79e0f80-kube-api-access-p6pbv\") pod \"migrator-66df5d496b-42qwv\" (UID: \"2a0b0e80-8291-4adf-83a3-7732f79e0f80\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-42qwv" Oct 30 14:06:45.662665 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.662540 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw84x\" (UniqueName: \"kubernetes.io/projected/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kube-api-access-zw84x\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.663017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.662993 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqwzb\" (UniqueName: \"kubernetes.io/projected/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-api-access-gqwzb\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.663108 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.663068 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmtf\" (UniqueName: \"kubernetes.io/projected/4171ff7a-5894-49d1-8683-606723fccdf9-kube-api-access-ssmtf\") pod \"insights-operator-7c4986d449-tfvs9\" (UID: \"4171ff7a-5894-49d1-8683-606723fccdf9\") " pod="openshift-insights/insights-operator-7c4986d449-tfvs9" Oct 30 14:06:45.663348 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.663323 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2zxn\" (UniqueName: \"kubernetes.io/projected/66b11a57-8746-4a93-b726-a877c2a22a98-kube-api-access-l2zxn\") pod \"console-operator-7cd56b8dbc-84mcb\" (UID: \"66b11a57-8746-4a93-b726-a877c2a22a98\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:45.663853 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.663623 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmq8x\" (UniqueName: \"kubernetes.io/projected/4f07e86b-0018-4c02-a9c6-cc24bfecd068-kube-api-access-wmq8x\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.663853 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.663801 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjm94\" (UniqueName: \"kubernetes.io/projected/78841e01-5bf7-4348-9802-09a81adeb88a-kube-api-access-xjm94\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:45.664082 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.663863 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-928ww\" (UniqueName: \"kubernetes.io/projected/fb9d3ab7-8a7f-4ccd-a336-912b00332110-kube-api-access-928ww\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-892zh\" (UID: \"fb9d3ab7-8a7f-4ccd-a336-912b00332110\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" Oct 30 14:06:45.664082 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.664021 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-bound-sa-token\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.664082 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.664058 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqjv\" (UniqueName: \"kubernetes.io/projected/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-kube-api-access-7sqjv\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:45.664246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.664136 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxk4p\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-kube-api-access-hxk4p\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.664246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.664233 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thjb5\" (UniqueName: \"kubernetes.io/projected/4070c760-6728-4166-a333-77af48851fd8-kube-api-access-thjb5\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.664388 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv27n\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-kube-api-access-jv27n\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.664465 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztlh\" (UniqueName: \"kubernetes.io/projected/fe78a7b6-cba1-4c52-b190-0febeaab5035-kube-api-access-lztlh\") pod \"network-check-source-6f78d59f99-2sxp4\" (UID: \"fe78a7b6-cba1-4c52-b190-0febeaab5035\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2sxp4" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.664514 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bmb\" (UniqueName: \"kubernetes.io/projected/17aed83f-3fbe-4b2e-8714-e61122539496-kube-api-access-82bmb\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.664571 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgn77\" (UniqueName: \"kubernetes.io/projected/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-kube-api-access-cgn77\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.664838 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-bound-sa-token\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.664825 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjlh\" (UniqueName: \"kubernetes.io/projected/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-kube-api-access-9gjlh\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.664906 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pwr6\" (UniqueName: \"kubernetes.io/projected/70ab068a-ed6b-4fae-8019-2e7c5873ab43-kube-api-access-9pwr6\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.665048 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7rf\" (UniqueName: \"kubernetes.io/projected/3829121e-3790-47ff-b827-8774a70fec04-kube-api-access-7w7rf\") pod \"kas-connection-checker-685895c7c9-dmj28\" (UID: \"3829121e-3790-47ff-b827-8774a70fec04\") " pod="kube-system/kas-connection-checker-685895c7c9-dmj28" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.665051 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhlw\" (UniqueName: \"kubernetes.io/projected/5b388277-9bf2-45be-b85d-2eafd1a1c543-kube-api-access-5xhlw\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.665092 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77kg8\" (UniqueName: \"kubernetes.io/projected/dfc6f25a-6a7a-472c-af78-683613ffb7a5-kube-api-access-77kg8\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.665153 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrj9\" (UniqueName: \"kubernetes.io/projected/42511697-bcd4-4df3-819c-ab38ba5159c3-kube-api-access-rrrj9\") pod \"service-ca-operator-544d749c8c-zvbvz\" (UID: \"42511697-bcd4-4df3-819c-ab38ba5159c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.665414 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbcd9\" (UniqueName: \"kubernetes.io/projected/1cdffa3f-7378-4f50-b1bc-e87aa647c4d8-kube-api-access-tbcd9\") pod \"kas-connection-checker-685895c7c9-jmrnq\" (UID: \"1cdffa3f-7378-4f50-b1bc-e87aa647c4d8\") " pod="kube-system/kas-connection-checker-685895c7c9-jmrnq" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.665481 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56lth\" (UniqueName: \"kubernetes.io/projected/a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5-kube-api-access-56lth\") pod \"kas-connection-checker-685895c7c9-f298s\" (UID: \"a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5\") " pod="kube-system/kas-connection-checker-685895c7c9-f298s" Oct 30 14:06:45.665531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.665503 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qdqj\" (UniqueName: \"kubernetes.io/projected/3a8ec17f-ff03-4af2-8885-1707bf84ec85-kube-api-access-8qdqj\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:45.666091 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.665826 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62nk\" (UniqueName: \"kubernetes.io/projected/03049b65-310d-4ad7-b614-1daf3a50b528-kube-api-access-f62nk\") pod \"volume-data-source-validator-58c9f5f46d-nw947\" (UID: \"03049b65-310d-4ad7-b614-1daf3a50b528\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nw947" Oct 30 14:06:45.670642 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.670620 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vklh\" (UniqueName: \"kubernetes.io/projected/911b39fc-f2e6-4094-9be0-68606d2aa4f1-kube-api-access-5vklh\") pod \"ingress-canary-m8jt7\" (UID: \"911b39fc-f2e6-4094-9be0-68606d2aa4f1\") " pod="openshift-ingress-canary/ingress-canary-m8jt7" Oct 30 14:06:45.670706 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.670617 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vhn\" (UniqueName: \"kubernetes.io/projected/aea273bc-03a8-41ad-8ed0-c1eab9a23a45-kube-api-access-f7vhn\") pod \"network-check-target-stdbh\" (UID: \"aea273bc-03a8-41ad-8ed0-c1eab9a23a45\") " pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:45.739349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739274 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-dbus\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:45.739349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739306 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-systemd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.739349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739332 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-ovn\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739356 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-etc-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739380 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739403 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-socket-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739415 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-etc-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739403 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-ovn\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739410 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-dbus\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739404 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-systemd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739427 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-conf\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739448 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-socket-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739429 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739499 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739525 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739549 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-systemd\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739558 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-conf\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739565 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739581 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-node-log\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.739582 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.739593 4968 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739603 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-kubernetes\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739612 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-systemd\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739618 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-host\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739623 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739652 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739655 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-host\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.739669 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls podName:4070c760-6728-4166-a333-77af48851fd8 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:46.23965445 +0000 UTC m=+1.865097820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zrb4p" (UID: "4070c760-6728-4166-a333-77af48851fd8") : secret "samples-operator-tls" not found Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739695 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-node-log\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739693 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-kubernetes\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.739703 4968 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739736 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-etc-kubernetes\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.739747 4968 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fff9cc46-btf9d: secret "image-registry-tls" not found Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739770 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-netd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.739801 4968 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739774 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-etc-kubernetes\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.739849 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls podName:bd9fc616-08fa-480b-bd77-622e4dfbf59e nodeName:}" failed. No retries permitted until 2026-10-30 14:06:46.239827666 +0000 UTC m=+1.865271046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls") pod "image-registry-8fff9cc46-btf9d" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e") : secret "image-registry-tls" not found Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739876 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-system-cni-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739873 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-netd\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739896 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-system-cni-dir\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739915 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cnibin\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.739942 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81e6ce71-7fb4-40ab-b14a-54457b97bd6a-cnibin\") pod \"multus-additional-cni-plugins-mssbd\" (UID: \"81e6ce71-7fb4-40ab-b14a-54457b97bd6a\") " pod="openshift-multus/multus-additional-cni-plugins-mssbd" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.739944 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:46.239920916 +0000 UTC m=+1.865364280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : secret "insights-runtime-extractor-tls" not found Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740013 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-systemd-units\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740030 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-hostroot\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740046 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740069 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740090 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-systemd-units\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740102 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-hostroot\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740121 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/354da4d4-65ce-4f03-a9aa-f7840bf28711-host\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.740128 4968 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740143 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740159 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-lib-modules\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.740167 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls podName:08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:46.240158702 +0000 UTC m=+1.865602054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rt4nm" (UID: "08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65") : secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740168 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/354da4d4-65ce-4f03-a9aa-f7840bf28711-host\") pod \"node-ca-vpkj5\" (UID: \"354da4d4-65ce-4f03-a9aa-f7840bf28711\") " pod="openshift-image-registry/node-ca-vpkj5" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740129 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740195 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.740220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740240 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-registration-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740256 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-kubelet-config\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740270 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-lib-modules\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740290 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a8ec17f-ff03-4af2-8885-1707bf84ec85-host-slash\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740302 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740317 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a8ec17f-ff03-4af2-8885-1707bf84ec85-host-slash\") pod \"iptables-alerter-xf6z2\" (UID: \"3a8ec17f-ff03-4af2-8885-1707bf84ec85\") " pod="openshift-network-operator/iptables-alerter-xf6z2" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740319 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740345 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-system-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740371 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740382 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6ae2497-ab63-4964-940a-e870d9131ae3-kubelet-config\") pod \"global-pull-secret-syncer-vlmdd\" (UID: \"f6ae2497-ab63-4964-940a-e870d9131ae3\") " pod="kube-system/global-pull-secret-syncer-vlmdd" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740347 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-registration-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.740399 4968 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740396 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-socket-dir-parent\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740427 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-system-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.740440 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls podName:17aed83f-3fbe-4b2e-8714-e61122539496 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:46.240428378 +0000 UTC m=+1.865871746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls") pod "dns-default-fvkvc" (UID: "17aed83f-3fbe-4b2e-8714-e61122539496") : secret "dns-default-metrics-tls" not found Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740448 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-run-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740458 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740481 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-var-lib-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740495 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-socket-dir-parent\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.740516 4968 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.740528 4968 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89594b78c-qxvnh: secret "image-registry-tls" not found Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740575 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-var-lib-openvswitch\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740588 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.740627 4968 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740624 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-netns\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.740662 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert podName:c2536dd8-0f2c-4571-b0db-484894b2a99f nodeName:}" failed. No retries permitted until 2026-10-30 14:06:46.240650584 +0000 UTC m=+1.866093939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9s2h" (UID: "c2536dd8-0f2c-4571-b0db-484894b2a99f") : secret "networking-console-plugin-cert" not found Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.740683 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls podName:fd07a434-a5f6-4cda-a93d-8824978aa590 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:46.240676084 +0000 UTC m=+1.866119435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls") pod "image-registry-89594b78c-qxvnh" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590") : secret "image-registry-tls" not found Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740698 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-netns\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740702 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-slash\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740745 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-slash\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740754 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-bin\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740783 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740796 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-cnibin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740818 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78841e01-5bf7-4348-9802-09a81adeb88a-hosts-file\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740827 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-cni-bin\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740837 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740868 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-sys\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740876 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-cnibin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.740894 4968 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740896 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-kubelet\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740923 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-crio-socket\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740928 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78841e01-5bf7-4348-9802-09a81adeb88a-hosts-file\") pod \"node-resolver-qnfg6\" (UID: \"78841e01-5bf7-4348-9802-09a81adeb88a\") " pod="openshift-dns/node-resolver-qnfg6" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740933 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-sys\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.740947 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:46.240932703 +0000 UTC m=+1.866376084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : secret "router-metrics-certs-default" not found Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740956 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-kubelet\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740984 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-crio-socket\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.740992 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-sys-fs\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741013 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-netns\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741028 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-k8s-cni-cncf-io\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741048 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-sys-fs\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741054 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-kubelet\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741064 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-host-run-netns\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741079 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-k8s-cni-cncf-io\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741069 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-conf-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741094 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-kubelet\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741105 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-conf-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741118 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741137 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741151 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysconfig\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741165 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-device-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741187 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741220 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-modprobe-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741237 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741240 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-device-dir\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.741249 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:46.241237523 +0000 UTC m=+1.866680892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : configmap references non-existent config key: service-ca.crt Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741241 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f753bb3b-c84d-4bd4-8b74-8ec50bb93c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-m27jb\" (UID: \"f753bb3b-c84d-4bd4-8b74-8ec50bb93c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m27jb" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741245 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-multus-cni-dir\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741276 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysconfig\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741309 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741318 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-modprobe-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741323 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-etc-sysctl-d\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741326 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-bin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741357 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-bin\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741353 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-run\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.741375 4968 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741387 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-multus-certs\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:45.741405 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs podName:a14b5c29-8cac-42f8-a3b2-6bf636cc05a4 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:46.241397217 +0000 UTC m=+1.866840580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs") pod "network-metrics-daemon-rgjxr" (UID: "a14b5c29-8cac-42f8-a3b2-6bf636cc05a4") : secret "metrics-daemon-secret" not found Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741424 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-log-socket\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741422 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-run-multus-certs\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741442 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-run\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741443 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-multus\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741476 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-var-lib-kubelet\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741483 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70ab068a-ed6b-4fae-8019-2e7c5873ab43-log-socket\") pod \"ovnkube-node-clqhk\" (UID: \"70ab068a-ed6b-4fae-8019-2e7c5873ab43\") " pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741509 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b388277-9bf2-45be-b85d-2eafd1a1c543-host-var-lib-cni-multus\") pod \"multus-fp57b\" (UID: \"5b388277-9bf2-45be-b85d-2eafd1a1c543\") " pod="openshift-multus/multus-fp57b" Oct 30 14:06:45.741544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.741543 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f07e86b-0018-4c02-a9c6-cc24bfecd068-var-lib-kubelet\") pod \"tuned-crmmp\" (UID: \"4f07e86b-0018-4c02-a9c6-cc24bfecd068\") " pod="openshift-cluster-node-tuning-operator/tuned-crmmp" Oct 30 14:06:45.812428 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.812403 4968 scope.go:122] "RemoveContainer" containerID="6f398efb0490e4b545d0c2f056c5615256be15aa38b06eb15dad2a77ff08d592" Oct 30 14:06:45.843318 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:45.843289 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:46.245634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.245595 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:46.245634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.245633 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:46.245860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.245658 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:46.245860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.245691 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:46.245860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.245744 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:46.245860 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245756 4968 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 14:06:46.245860 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245776 4968 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:46.245860 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245794 4968 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fff9cc46-btf9d: secret "image-registry-tls" not found Oct 30 14:06:46.245860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.245776 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:46.245860 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245822 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls podName:4070c760-6728-4166-a333-77af48851fd8 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.245803654 +0000 UTC m=+2.871247012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zrb4p" (UID: "4070c760-6728-4166-a333-77af48851fd8") : secret "samples-operator-tls" not found Oct 30 14:06:46.245860 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245835 4968 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:46.245860 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245845 4968 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89594b78c-qxvnh: secret "image-registry-tls" not found Oct 30 14:06:46.245860 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245849 4968 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245881 4968 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245855 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls podName:bd9fc616-08fa-480b-bd77-622e4dfbf59e nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.245839123 +0000 UTC m=+2.871282488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls") pod "image-registry-8fff9cc46-btf9d" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e") : secret "image-registry-tls" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.245923 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245899 4968 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245950 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls podName:17aed83f-3fbe-4b2e-8714-e61122539496 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.245934449 +0000 UTC m=+2.871377843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls") pod "dns-default-fvkvc" (UID: "17aed83f-3fbe-4b2e-8714-e61122539496") : secret "dns-default-metrics-tls" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245978 4968 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.245983 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls podName:fd07a434-a5f6-4cda-a93d-8824978aa590 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.245971625 +0000 UTC m=+2.871414993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls") pod "image-registry-89594b78c-qxvnh" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590") : secret "image-registry-tls" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.246011 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.24599443 +0000 UTC m=+2.871437795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : secret "insights-runtime-extractor-tls" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.246032 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls podName:08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.246022606 +0000 UTC m=+2.871465956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rt4nm" (UID: "08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65") : secret "cluster-monitoring-operator-tls" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.246050 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert podName:c2536dd8-0f2c-4571-b0db-484894b2a99f nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.246040842 +0000 UTC m=+2.871484231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9s2h" (UID: "c2536dd8-0f2c-4571-b0db-484894b2a99f") : secret "networking-console-plugin-cert" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.246092 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.246137 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.246194 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.246217 4968 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.246261 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.246249008 +0000 UTC m=+2.871692362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : secret "router-metrics-certs-default" not found Oct 30 14:06:46.246317 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.246303 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.246288842 +0000 UTC m=+2.871732203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : configmap references non-existent config key: service-ca.crt Oct 30 14:06:46.246787 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.246330 4968 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 14:06:46.246787 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.246364 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs podName:a14b5c29-8cac-42f8-a3b2-6bf636cc05a4 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:47.246351788 +0000 UTC m=+2.871795182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs") pod "network-metrics-daemon-rgjxr" (UID: "a14b5c29-8cac-42f8-a3b2-6bf636cc05a4") : secret "metrics-daemon-secret" not found Oct 30 14:06:46.572467 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.572390 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:06:46.572917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.572830 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/1.log" Oct 30 14:06:46.572917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.572867 4968 generic.go:356] "Generic (PLEG): container finished" podID="66b11a57-8746-4a93-b726-a877c2a22a98" containerID="8e04049b8876eeb42c972d4505b4fdf39c687c5ad5a85486f6409204e6d1bf8e" exitCode=255 Oct 30 14:06:46.573013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.572948 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" event={"ID":"66b11a57-8746-4a93-b726-a877c2a22a98","Type":"ContainerDied","Data":"8e04049b8876eeb42c972d4505b4fdf39c687c5ad5a85486f6409204e6d1bf8e"} Oct 30 14:06:46.573013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.573003 4968 scope.go:122] "RemoveContainer" containerID="6f398efb0490e4b545d0c2f056c5615256be15aa38b06eb15dad2a77ff08d592" Oct 30 14:06:46.573438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:46.573417 4968 scope.go:122] "RemoveContainer" containerID="8e04049b8876eeb42c972d4505b4fdf39c687c5ad5a85486f6409204e6d1bf8e" Oct 30 14:06:46.573605 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:46.573583 4968 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-84mcb_openshift-console-operator(66b11a57-8746-4a93-b726-a877c2a22a98)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" podUID="66b11a57-8746-4a93-b726-a877c2a22a98" Oct 30 14:06:47.256653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.256615 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:47.256653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.256656 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.256677 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.256697 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.256740 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.256765 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.256795 4968 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.256846 4968 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.256868 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls podName:17aed83f-3fbe-4b2e-8714-e61122539496 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.256848196 +0000 UTC m=+4.882291564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls") pod "dns-default-fvkvc" (UID: "17aed83f-3fbe-4b2e-8714-e61122539496") : secret "dns-default-metrics-tls" not found Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.256882 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.256876627 +0000 UTC m=+4.882319981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : configmap references non-existent config key: service-ca.crt Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.256850 4968 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.256892 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.256887431 +0000 UTC m=+4.882330782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : secret "router-metrics-certs-default" not found Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.256913 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs podName:a14b5c29-8cac-42f8-a3b2-6bf636cc05a4 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.256901812 +0000 UTC m=+4.882345193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs") pod "network-metrics-daemon-rgjxr" (UID: "a14b5c29-8cac-42f8-a3b2-6bf636cc05a4") : secret "metrics-daemon-secret" not found Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.256799 4968 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.256929 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.256954 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.256934 4968 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89594b78c-qxvnh: secret "image-registry-tls" not found Oct 30 14:06:47.256992 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.256990 4968 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.256795 4968 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.257023 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls podName:4070c760-6728-4166-a333-77af48851fd8 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.257012508 +0000 UTC m=+4.882455873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zrb4p" (UID: "4070c760-6728-4166-a333-77af48851fd8") : secret "samples-operator-tls" not found Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.256980 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.257037 4968 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.257068 4968 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.257080 4968 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fff9cc46-btf9d: secret "image-registry-tls" not found Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.257041 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls podName:fd07a434-a5f6-4cda-a93d-8824978aa590 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.257030554 +0000 UTC m=+4.882473941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls") pod "image-registry-89594b78c-qxvnh" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590") : secret "image-registry-tls" not found Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.257123 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert podName:c2536dd8-0f2c-4571-b0db-484894b2a99f nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.257112371 +0000 UTC m=+4.882555730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9s2h" (UID: "c2536dd8-0f2c-4571-b0db-484894b2a99f") : secret "networking-console-plugin-cert" not found Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.257135 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.257130189 +0000 UTC m=+4.882573540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : secret "insights-runtime-extractor-tls" not found Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.257166 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls podName:bd9fc616-08fa-480b-bd77-622e4dfbf59e nodeName:}" failed. No retries permitted until 2026-10-30 14:06:49.257160612 +0000 UTC m=+4.882603967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls") pod "image-registry-8fff9cc46-btf9d" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e") : secret "image-registry-tls" not found Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.257201 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:47.257598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.257498 4968 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 14:06:47.259511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.259496 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rt4nm\" (UID: \"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:47.313079 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.313056 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" Oct 30 14:06:47.317794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.317775 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:47.577919 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.577846 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:06:47.578382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.578286 4968 scope.go:122] "RemoveContainer" containerID="8e04049b8876eeb42c972d4505b4fdf39c687c5ad5a85486f6409204e6d1bf8e" Oct 30 14:06:47.582672 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:47.582633 4968 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-84mcb_openshift-console-operator(66b11a57-8746-4a93-b726-a877c2a22a98)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" podUID="66b11a57-8746-4a93-b726-a877c2a22a98" Oct 30 14:06:47.609742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.609690 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm"] Oct 30 14:06:47.614672 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:47.614640 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b6ecf0_e4e9_46ce_a2ae_6a768ce8be65.slice/crio-cf06eff9c0afb8696080e3852275325fee94a205e0a2acaa7d4a8084ee76241f WatchSource:0}: Error finding container cf06eff9c0afb8696080e3852275325fee94a205e0a2acaa7d4a8084ee76241f: Status 404 returned error can't find the container with id cf06eff9c0afb8696080e3852275325fee94a205e0a2acaa7d4a8084ee76241f Oct 30 14:06:47.616602 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:47.616584 4968 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:06:48.582882 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:48.582842 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" event={"ID":"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65","Type":"ContainerStarted","Data":"cf06eff9c0afb8696080e3852275325fee94a205e0a2acaa7d4a8084ee76241f"} Oct 30 14:06:49.278820 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.278782 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:49.278820 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.278822 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:49.279037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.278849 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:49.279037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.278898 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:49.279037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.278930 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:49.279037 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:49.278939 4968 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 14:06:49.279037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.278955 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:49.279037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.278989 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:49.279037 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:49.279013 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls podName:4070c760-6728-4166-a333-77af48851fd8 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:53.278993669 +0000 UTC m=+8.904437034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zrb4p" (UID: "4070c760-6728-4166-a333-77af48851fd8") : secret "samples-operator-tls" not found Oct 30 14:06:49.279393 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:49.279040 4968 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 14:06:49.279393 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.279066 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:49.279393 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:49.279105 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:53.279087758 +0000 UTC m=+8.904531111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : secret "insights-runtime-extractor-tls" not found Oct 30 14:06:49.279393 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:49.279051 4968 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 14:06:49.279393 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:49.279146 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle podName:dfc6f25a-6a7a-472c-af78-683613ffb7a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:53.279135069 +0000 UTC m=+8.904578469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle") pod "router-default-598ddffcff-nctk8" (UID: "dfc6f25a-6a7a-472c-af78-683613ffb7a5") : configmap references non-existent config key: service-ca.crt Oct 30 14:06:49.279393 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.279168 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:49.279393 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:49.279216 4968 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 14:06:49.279393 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:49.279249 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs podName:a14b5c29-8cac-42f8-a3b2-6bf636cc05a4 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:53.279236961 +0000 UTC m=+8.904680316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs") pod "network-metrics-daemon-rgjxr" (UID: "a14b5c29-8cac-42f8-a3b2-6bf636cc05a4") : secret "metrics-daemon-secret" not found Oct 30 14:06:49.279393 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:49.279266 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls podName:17aed83f-3fbe-4b2e-8714-e61122539496 nodeName:}" failed. No retries permitted until 2026-10-30 14:06:53.279255027 +0000 UTC m=+8.904698414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls") pod "dns-default-fvkvc" (UID: "17aed83f-3fbe-4b2e-8714-e61122539496") : secret "dns-default-metrics-tls" not found Oct 30 14:06:49.280110 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:49.279415 4968 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 14:06:49.280110 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:49.279466 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert podName:c2536dd8-0f2c-4571-b0db-484894b2a99f nodeName:}" failed. No retries permitted until 2026-10-30 14:06:53.279449893 +0000 UTC m=+8.904893252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9s2h" (UID: "c2536dd8-0f2c-4571-b0db-484894b2a99f") : secret "networking-console-plugin-cert" not found Oct 30 14:06:49.281447 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.281424 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"image-registry-8fff9cc46-btf9d\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:49.281932 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.281913 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc6f25a-6a7a-472c-af78-683613ffb7a5-metrics-certs\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:49.282001 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.281915 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"image-registry-89594b78c-qxvnh\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:49.358652 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.358625 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:06:49.358879 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.358858 4968 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 14:06:49.411696 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.411657 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:49.424063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.423587 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:49.588905 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.588878 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rt4nm" event={"ID":"08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65","Type":"ContainerStarted","Data":"fa2234ff8c9c9d070f2ac89095eb1742010d527ca6885dd68ec28c7e7b601d12"} Oct 30 14:06:49.725432 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.725402 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8fff9cc46-btf9d"] Oct 30 14:06:49.731104 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:49.731076 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9fc616_08fa_480b_bd77_622e4dfbf59e.slice/crio-c57e831baea96c37f590c0740af97f38e25132459d24ea72344781b3bca30aa0 WatchSource:0}: Error finding container c57e831baea96c37f590c0740af97f38e25132459d24ea72344781b3bca30aa0: Status 404 returned error can't find the container with id c57e831baea96c37f590c0740af97f38e25132459d24ea72344781b3bca30aa0 Oct 30 14:06:49.741990 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:49.741966 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-89594b78c-qxvnh"] Oct 30 14:06:49.747410 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:49.747384 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd07a434_a5f6_4cda_a93d_8824978aa590.slice/crio-41c742342fa919c34077244f1e26e10c7bee69f9f6eea40a14950048fe1bd975 WatchSource:0}: Error finding container 41c742342fa919c34077244f1e26e10c7bee69f9f6eea40a14950048fe1bd975: Status 404 returned error can't find the container with id 41c742342fa919c34077244f1e26e10c7bee69f9f6eea40a14950048fe1bd975 Oct 30 14:06:50.596993 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:50.596958 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" event={"ID":"fd07a434-a5f6-4cda-a93d-8824978aa590","Type":"ContainerStarted","Data":"0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203"} Oct 30 14:06:50.597501 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:50.597044 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" event={"ID":"fd07a434-a5f6-4cda-a93d-8824978aa590","Type":"ContainerStarted","Data":"41c742342fa919c34077244f1e26e10c7bee69f9f6eea40a14950048fe1bd975"} Oct 30 14:06:50.598524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:50.598497 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" event={"ID":"bd9fc616-08fa-480b-bd77-622e4dfbf59e","Type":"ContainerStarted","Data":"2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1"} Oct 30 14:06:50.598637 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:50.598543 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" event={"ID":"bd9fc616-08fa-480b-bd77-622e4dfbf59e","Type":"ContainerStarted","Data":"c57e831baea96c37f590c0740af97f38e25132459d24ea72344781b3bca30aa0"} Oct 30 14:06:50.805625 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:50.805600 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qnfg6_78841e01-5bf7-4348-9802-09a81adeb88a/dns-node-resolver/0.log" Oct 30 14:06:51.206278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:51.206247 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-89594b78c-qxvnh_fd07a434-a5f6-4cda-a93d-8824978aa590/registry/0.log" Oct 30 14:06:51.405753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:51.405706 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8fff9cc46-btf9d_bd9fc616-08fa-480b-bd77-622e4dfbf59e/registry/0.log" Oct 30 14:06:51.601605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:51.601574 4968 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 14:06:51.602107 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:51.601574 4968 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 14:06:52.005270 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:52.005201 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vpkj5_354da4d4-65ce-4f03-a9aa-f7840bf28711/node-ca/0.log" Oct 30 14:06:52.605736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:52.605696 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m8jt7_911b39fc-f2e6-4094-9be0-68606d2aa4f1/serve-healthcheck-canary/0.log" Oct 30 14:06:52.806384 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:52.806359 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-42qwv_2a0b0e80-8291-4adf-83a3-7732f79e0f80/migrator/0.log" Oct 30 14:06:53.206605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.206571 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-892zh_fb9d3ab7-8a7f-4ccd-a336-912b00332110/kube-storage-version-migrator-operator/0.log" Oct 30 14:06:53.320495 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.320459 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:06:53.320655 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.320505 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:53.320655 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.320548 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:53.320655 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.320588 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:53.320655 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:53.320611 4968 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 14:06:53.320655 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.320640 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:53.320970 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:53.320676 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls podName:17aed83f-3fbe-4b2e-8714-e61122539496 nodeName:}" failed. No retries permitted until 2026-10-30 14:07:01.3206567 +0000 UTC m=+16.946100070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls") pod "dns-default-fvkvc" (UID: "17aed83f-3fbe-4b2e-8714-e61122539496") : secret "dns-default-metrics-tls" not found Oct 30 14:06:53.320970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.320732 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:06:53.320970 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:53.320836 4968 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 14:06:53.320970 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:53.320877 4968 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls podName:2e3b429d-b5ef-40bf-b31a-9bb2b5085010 nodeName:}" failed. No retries permitted until 2026-10-30 14:07:01.320863405 +0000 UTC m=+16.946306759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmtlt" (UID: "2e3b429d-b5ef-40bf-b31a-9bb2b5085010") : secret "insights-runtime-extractor-tls" not found Oct 30 14:06:53.321200 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.321184 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc6f25a-6a7a-472c-af78-683613ffb7a5-service-ca-bundle\") pod \"router-default-598ddffcff-nctk8\" (UID: \"dfc6f25a-6a7a-472c-af78-683613ffb7a5\") " pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:53.322988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.322966 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14b5c29-8cac-42f8-a3b2-6bf636cc05a4-metrics-certs\") pod \"network-metrics-daemon-rgjxr\" (UID: \"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4\") " pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:53.323096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.323044 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2536dd8-0f2c-4571-b0db-484894b2a99f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9s2h\" (UID: \"c2536dd8-0f2c-4571-b0db-484894b2a99f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:53.323096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.323072 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4070c760-6728-4166-a333-77af48851fd8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zrb4p\" (UID: \"4070c760-6728-4166-a333-77af48851fd8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:53.610880 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.610857 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:53.613998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.613979 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" Oct 30 14:06:53.614067 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.614036 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" Oct 30 14:06:53.617680 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.617661 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgjxr" Oct 30 14:06:53.929621 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.929547 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p"] Oct 30 14:06:53.941146 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.941124 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rgjxr"] Oct 30 14:06:53.944350 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.944327 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-598ddffcff-nctk8"] Oct 30 14:06:53.948882 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:53.948846 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14b5c29_8cac_42f8_a3b2_6bf636cc05a4.slice/crio-9720b09810646811501e0c0fcb7ecdd599016cd4554f34836907a526798ddbca WatchSource:0}: Error finding container 9720b09810646811501e0c0fcb7ecdd599016cd4554f34836907a526798ddbca: Status 404 returned error can't find the container with id 9720b09810646811501e0c0fcb7ecdd599016cd4554f34836907a526798ddbca Oct 30 14:06:53.949119 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:53.949097 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc6f25a_6a7a_472c_af78_683613ffb7a5.slice/crio-8d7673bb2a5ccf7fc8c7a1f816f1f7a45c43140cfb5c85dc23649e7908c8233c WatchSource:0}: Error finding container 8d7673bb2a5ccf7fc8c7a1f816f1f7a45c43140cfb5c85dc23649e7908c8233c: Status 404 returned error can't find the container with id 8d7673bb2a5ccf7fc8c7a1f816f1f7a45c43140cfb5c85dc23649e7908c8233c Oct 30 14:06:53.950893 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:53.950873 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-w9s2h"] Oct 30 14:06:53.955730 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:06:53.955693 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2536dd8_0f2c_4571_b0db_484894b2a99f.slice/crio-00c173087c9152e326ccc78d4350fc61fc487ed4d5406caa31fbca8f45149d3c WatchSource:0}: Error finding container 00c173087c9152e326ccc78d4350fc61fc487ed4d5406caa31fbca8f45149d3c: Status 404 returned error can't find the container with id 00c173087c9152e326ccc78d4350fc61fc487ed4d5406caa31fbca8f45149d3c Oct 30 14:06:54.109017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:54.108983 4968 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:54.109406 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:54.109388 4968 scope.go:122] "RemoveContainer" containerID="8e04049b8876eeb42c972d4505b4fdf39c687c5ad5a85486f6409204e6d1bf8e" Oct 30 14:06:54.109623 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:54.109602 4968 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-84mcb_openshift-console-operator(66b11a57-8746-4a93-b726-a877c2a22a98)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" podUID="66b11a57-8746-4a93-b726-a877c2a22a98" Oct 30 14:06:54.612677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:54.612643 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rgjxr" event={"ID":"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4","Type":"ContainerStarted","Data":"9720b09810646811501e0c0fcb7ecdd599016cd4554f34836907a526798ddbca"} Oct 30 14:06:54.613925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:54.613896 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" event={"ID":"4070c760-6728-4166-a333-77af48851fd8","Type":"ContainerStarted","Data":"a21673fe6a81c132e6310bd6501b560726640ebe5fd5723aa4f4f7f6e6c0af03"} Oct 30 14:06:54.615683 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:54.615654 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-598ddffcff-nctk8" event={"ID":"dfc6f25a-6a7a-472c-af78-683613ffb7a5","Type":"ContainerStarted","Data":"91ee136094fba88c49734dfb5bc07780ea50f4b2845b1d2753bf82d2d0323382"} Oct 30 14:06:54.615836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:54.615700 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-598ddffcff-nctk8" event={"ID":"dfc6f25a-6a7a-472c-af78-683613ffb7a5","Type":"ContainerStarted","Data":"8d7673bb2a5ccf7fc8c7a1f816f1f7a45c43140cfb5c85dc23649e7908c8233c"} Oct 30 14:06:54.617333 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:54.617309 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" event={"ID":"c2536dd8-0f2c-4571-b0db-484894b2a99f","Type":"ContainerStarted","Data":"00c173087c9152e326ccc78d4350fc61fc487ed4d5406caa31fbca8f45149d3c"} Oct 30 14:06:54.922024 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:54.921941 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:55.046912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.046879 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:06:55.185772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.185738 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:06:55.586962 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.586938 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:06:55.611764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.611742 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:55.614801 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.614781 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:55.624361 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.624200 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-w9s2h" event={"ID":"c2536dd8-0f2c-4571-b0db-484894b2a99f","Type":"ContainerStarted","Data":"2cbd31c360a4b4ef3956c67c0aa334d1897cf84421e8f83ed4106caa2978228b"} Oct 30 14:06:55.626869 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.626842 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rgjxr" event={"ID":"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4","Type":"ContainerStarted","Data":"3146328764be66fce2eb0934cabc6e051aa6a23e61abb5c11d22532dacf78bdc"} Oct 30 14:06:55.626965 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.626887 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rgjxr" event={"ID":"a14b5c29-8cac-42f8-a3b2-6bf636cc05a4","Type":"ContainerStarted","Data":"0bb14439d1efef53bbde2a26a3344791e5ece6b76051bb7554f52b5002bd7972"} Oct 30 14:06:55.629022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.628997 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" event={"ID":"4070c760-6728-4166-a333-77af48851fd8","Type":"ContainerStarted","Data":"5501875416fc5ee63f8773f99c34491d48c7a02d8509a621270b1de47f6ef40b"} Oct 30 14:06:55.630493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.630472 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-598ddffcff-nctk8" Oct 30 14:06:55.843376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.843343 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:55.843786 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:55.843772 4968 scope.go:122] "RemoveContainer" containerID="8e04049b8876eeb42c972d4505b4fdf39c687c5ad5a85486f6409204e6d1bf8e" Oct 30 14:06:55.843971 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:06:55.843950 4968 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-84mcb_openshift-console-operator(66b11a57-8746-4a93-b726-a877c2a22a98)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" podUID="66b11a57-8746-4a93-b726-a877c2a22a98" Oct 30 14:06:56.634098 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:56.634054 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zrb4p" event={"ID":"4070c760-6728-4166-a333-77af48851fd8","Type":"ContainerStarted","Data":"c387b100794036045f37811eba03761a7cb83159bc3dd3d677414692fad2a63a"} Oct 30 14:06:56.634552 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:56.634423 4968 scope.go:122] "RemoveContainer" containerID="8e04049b8876eeb42c972d4505b4fdf39c687c5ad5a85486f6409204e6d1bf8e" Oct 30 14:06:57.639895 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:57.639868 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:06:57.640269 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:57.639985 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" event={"ID":"66b11a57-8746-4a93-b726-a877c2a22a98","Type":"ContainerStarted","Data":"444c3659366a77bdd1f97e22bcfd025fcdbba52ddb3cf615c4e7f7301f34704f"} Oct 30 14:06:57.640488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:57.640458 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:06:57.644982 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:06:57.644961 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-84mcb" Oct 30 14:07:01.388998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:01.388961 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:07:01.388998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:01.389006 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:07:01.391348 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:01.391317 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3b429d-b5ef-40bf-b31a-9bb2b5085010-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmtlt\" (UID: \"2e3b429d-b5ef-40bf-b31a-9bb2b5085010\") " pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:07:01.391473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:01.391359 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17aed83f-3fbe-4b2e-8714-e61122539496-metrics-tls\") pod \"dns-default-fvkvc\" (UID: \"17aed83f-3fbe-4b2e-8714-e61122539496\") " pod="openshift-dns/dns-default-fvkvc" Oct 30 14:07:01.423951 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:01.423922 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fvkvc" Oct 30 14:07:01.425000 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:01.424984 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hmtlt" Oct 30 14:07:01.759608 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:01.759524 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fvkvc"] Oct 30 14:07:01.764840 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:07:01.764813 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17aed83f_3fbe_4b2e_8714_e61122539496.slice/crio-2a572782d4fbae3708ab01cf3f814aea6e816ea14effdbff0588beedee8b9bd2 WatchSource:0}: Error finding container 2a572782d4fbae3708ab01cf3f814aea6e816ea14effdbff0588beedee8b9bd2: Status 404 returned error can't find the container with id 2a572782d4fbae3708ab01cf3f814aea6e816ea14effdbff0588beedee8b9bd2 Oct 30 14:07:01.767168 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:07:01.767144 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3b429d_b5ef_40bf_b31a_9bb2b5085010.slice/crio-cc4f2ff9833c3ed19f8df7c24ebe47aaa79372a0043f88f42b6aae3ac472eaf1 WatchSource:0}: Error finding container cc4f2ff9833c3ed19f8df7c24ebe47aaa79372a0043f88f42b6aae3ac472eaf1: Status 404 returned error can't find the container with id cc4f2ff9833c3ed19f8df7c24ebe47aaa79372a0043f88f42b6aae3ac472eaf1 Oct 30 14:07:01.769475 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:01.769454 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hmtlt"] Oct 30 14:07:02.494955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:02.494917 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8fff9cc46-btf9d"] Oct 30 14:07:02.659038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:02.659007 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fvkvc" event={"ID":"17aed83f-3fbe-4b2e-8714-e61122539496","Type":"ContainerStarted","Data":"2a572782d4fbae3708ab01cf3f814aea6e816ea14effdbff0588beedee8b9bd2"} Oct 30 14:07:02.661154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:02.661125 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmtlt" event={"ID":"2e3b429d-b5ef-40bf-b31a-9bb2b5085010","Type":"ContainerStarted","Data":"31b46fe64e548bb6024cce1642969a9f9076b53e552dddc694a54218855cb6fe"} Oct 30 14:07:02.661262 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:02.661181 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmtlt" event={"ID":"2e3b429d-b5ef-40bf-b31a-9bb2b5085010","Type":"ContainerStarted","Data":"d8558518147ddf900450a50e8052e478a2539fccb5013e0e0af932713a44d6c3"} Oct 30 14:07:02.661262 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:02.661201 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmtlt" event={"ID":"2e3b429d-b5ef-40bf-b31a-9bb2b5085010","Type":"ContainerStarted","Data":"cc4f2ff9833c3ed19f8df7c24ebe47aaa79372a0043f88f42b6aae3ac472eaf1"} Oct 30 14:07:03.670600 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:03.670233 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fvkvc" event={"ID":"17aed83f-3fbe-4b2e-8714-e61122539496","Type":"ContainerStarted","Data":"68cff2a1c4c8b37360aa2e5db96e90e2545e1618de58c94f7beed3318457408a"} Oct 30 14:07:04.675824 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:04.675778 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fvkvc" event={"ID":"17aed83f-3fbe-4b2e-8714-e61122539496","Type":"ContainerStarted","Data":"09882d15d903032a68abefbadf1974e7543741d4d135df6b2fad64831c1146f8"} Oct 30 14:07:04.676281 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:04.675878 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fvkvc" Oct 30 14:07:04.677725 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:04.677686 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmtlt" event={"ID":"2e3b429d-b5ef-40bf-b31a-9bb2b5085010","Type":"ContainerStarted","Data":"53d529679835d26a52a13ddfa15a6a4bdabcb8789fa733965796099cb4419c8f"} Oct 30 14:07:05.052045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:05.051962 4968 patch_prober.go:29] interesting pod/image-registry-89594b78c-qxvnh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 14:07:05.052045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:05.052021 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" podUID="fd07a434-a5f6-4cda-a93d-8824978aa590" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:07:08.210315 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:08.210282 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h6hnr" Oct 30 14:07:08.918130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:08.918103 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clqhk" Oct 30 14:07:09.218428 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:09.218348 4968 patch_prober.go:29] interesting pod/image-registry-89594b78c-qxvnh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 14:07:09.218428 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:09.218400 4968 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" podUID="fd07a434-a5f6-4cda-a93d-8824978aa590" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:07:11.217064 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.217027 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x9ht5"] Oct 30 14:07:11.241853 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.241824 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.255413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.255383 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-tls\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.255552 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.255447 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-metrics-client-ca\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.255552 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.255513 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-wtmp\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.255663 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.255578 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.255663 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.255615 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4nr6\" (UniqueName: \"kubernetes.io/projected/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-kube-api-access-k4nr6\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.255793 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.255691 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-sys\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.255793 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.255729 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-root\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.255874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.255797 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.255874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.255832 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-textfile\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.294213 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.294180 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 14:07:11.294213 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.294210 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 14:07:11.294388 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.294276 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8kn7g\"" Oct 30 14:07:11.339173 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.339149 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 14:07:11.339173 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.339173 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 14:07:11.356645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.356615 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.356827 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.356650 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-textfile\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.356827 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.356708 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-tls\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.356943 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.356859 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-metrics-client-ca\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.356943 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.356904 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-wtmp\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.356943 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.356929 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.357096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.356963 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nr6\" (UniqueName: \"kubernetes.io/projected/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-kube-api-access-k4nr6\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.357096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.356989 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-sys\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.357096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.357009 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-root\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.357096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.357072 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-sys\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.357096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.357072 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-wtmp\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.357309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.357103 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-textfile\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.357309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.357151 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-root\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.357309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.357291 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.357433 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.357387 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-metrics-client-ca\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.359181 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.359158 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-tls\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.359282 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.359195 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.380609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.380590 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4nr6\" (UniqueName: \"kubernetes.io/projected/107b3cc3-f2aa-469f-b5a3-6379efd7a53a-kube-api-access-k4nr6\") pod \"node-exporter-x9ht5\" (UID: \"107b3cc3-f2aa-469f-b5a3-6379efd7a53a\") " pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.592998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.592967 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x9ht5" Oct 30 14:07:11.624842 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:07:11.624811 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107b3cc3_f2aa_469f_b5a3_6379efd7a53a.slice/crio-c754fbaa11b69a1bda257d1174fd64e8bc1671089dd1d2588eb35075442dfe27 WatchSource:0}: Error finding container c754fbaa11b69a1bda257d1174fd64e8bc1671089dd1d2588eb35075442dfe27: Status 404 returned error can't find the container with id c754fbaa11b69a1bda257d1174fd64e8bc1671089dd1d2588eb35075442dfe27 Oct 30 14:07:11.704996 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:11.704961 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9ht5" event={"ID":"107b3cc3-f2aa-469f-b5a3-6379efd7a53a","Type":"ContainerStarted","Data":"c754fbaa11b69a1bda257d1174fd64e8bc1671089dd1d2588eb35075442dfe27"} Oct 30 14:07:12.501245 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:12.501212 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:07:13.714602 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:13.714570 4968 generic.go:356] "Generic (PLEG): container finished" podID="107b3cc3-f2aa-469f-b5a3-6379efd7a53a" containerID="4c6b38a2834c498a40a09dc47e8ecf5b5ef6cd996734a102d97116ec14ad4052" exitCode=0 Oct 30 14:07:13.715010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:13.714632 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9ht5" event={"ID":"107b3cc3-f2aa-469f-b5a3-6379efd7a53a","Type":"ContainerDied","Data":"4c6b38a2834c498a40a09dc47e8ecf5b5ef6cd996734a102d97116ec14ad4052"} Oct 30 14:07:14.684322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:14.684295 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fvkvc" Oct 30 14:07:14.720659 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:14.720627 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9ht5" event={"ID":"107b3cc3-f2aa-469f-b5a3-6379efd7a53a","Type":"ContainerStarted","Data":"98cdb8edb35e3cf68b655552cc45f30d4a53f6992d0459e3f028eb65c89f5877"} Oct 30 14:07:14.721097 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:14.720701 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9ht5" event={"ID":"107b3cc3-f2aa-469f-b5a3-6379efd7a53a","Type":"ContainerStarted","Data":"4b1527d76b80d80ea9e71ab82d2d2660ed605f40637134975522f61d414cc2f6"} Oct 30 14:07:14.762831 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:14.762779 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x9ht5" podStartSLOduration=2.347572519 podStartE2EDuration="3.762767527s" totalImagesPullingTime="1.342451465s" totalInitContainerRuntime="72.743543ms" isStatefulPod=true podCreationTimestamp="2026-10-30 14:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:14.762137844 +0000 UTC m=+30.387581245" watchObservedRunningTime="2026-10-30 14:07:14.762767527 +0000 UTC m=+30.388210899" Oct 30 14:07:15.051322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:15.051249 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:07:15.590280 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:15.590250 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-stdbh" Oct 30 14:07:20.442902 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:07:20.442868 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:07:24.790704 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:24.790672 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-89594b78c-qxvnh"] Oct 30 14:07:27.518270 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.518174 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" podUID="bd9fc616-08fa-480b-bd77-622e4dfbf59e" containerName="registry" containerID="cri-o://2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1" gracePeriod=30 Oct 30 14:07:27.764158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.764134 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:07:27.773084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.773027 4968 generic.go:356] "Generic (PLEG): container finished" podID="bd9fc616-08fa-480b-bd77-622e4dfbf59e" containerID="2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1" exitCode=0 Oct 30 14:07:27.773084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.773062 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" Oct 30 14:07:27.773230 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.773109 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" event={"ID":"bd9fc616-08fa-480b-bd77-622e4dfbf59e","Type":"ContainerDied","Data":"2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1"} Oct 30 14:07:27.773230 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.773160 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fff9cc46-btf9d" event={"ID":"bd9fc616-08fa-480b-bd77-622e4dfbf59e","Type":"ContainerDied","Data":"c57e831baea96c37f590c0740af97f38e25132459d24ea72344781b3bca30aa0"} Oct 30 14:07:27.773230 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.773182 4968 scope.go:122] "RemoveContainer" containerID="2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1" Oct 30 14:07:27.781112 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.781096 4968 scope.go:122] "RemoveContainer" containerID="2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1" Oct 30 14:07:27.781404 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:07:27.781376 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1\": container with ID starting with 2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1 not found: ID does not exist" containerID="2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1" Oct 30 14:07:27.781457 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.781415 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1"} err="failed to get container status \"2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1\": rpc error: code = NotFound desc = could not find container \"2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1\": container with ID starting with 2b67e6e78b0ac7a0fb7ea454a9fd055febf0b8f223a209970f0113c982e9feb1 not found: ID does not exist" Oct 30 14:07:27.893797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.893767 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-installation-pull-secrets\") pod \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " Oct 30 14:07:27.893958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.893820 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-trusted-ca\") pod \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " Oct 30 14:07:27.893958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.893847 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9fc616-08fa-480b-bd77-622e4dfbf59e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd9fc616-08fa-480b-bd77-622e4dfbf59e-ca-trust-extracted\") pod \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " Oct 30 14:07:27.893958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.893866 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-bound-sa-token\") pod \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " Oct 30 14:07:27.893958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.893884 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-image-registry-private-configuration\") pod \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " Oct 30 14:07:27.893958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.893935 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-kube-api-access-hxk4p\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-kube-api-access-hxk4p\") pod \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " Oct 30 14:07:27.894187 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.893986 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") pod \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " Oct 30 14:07:27.894187 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.894043 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-certificates\") pod \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\" (UID: \"bd9fc616-08fa-480b-bd77-622e4dfbf59e\") " Oct 30 14:07:27.894361 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.894334 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-trusted-ca" pod "bd9fc616-08fa-480b-bd77-622e4dfbf59e" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:27.894858 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.894829 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-certificates" pod "bd9fc616-08fa-480b-bd77-622e4dfbf59e" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:27.896451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.896418 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls" pod "bd9fc616-08fa-480b-bd77-622e4dfbf59e" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:27.896541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.896429 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-installation-pull-secrets" pod "bd9fc616-08fa-480b-bd77-622e4dfbf59e" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:27.896794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.896773 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-kube-api-access-hxk4p" pod "bd9fc616-08fa-480b-bd77-622e4dfbf59e" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e"). InnerVolumeSpecName "kube-api-access-hxk4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:27.896867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.896815 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-image-registry-private-configuration" pod "bd9fc616-08fa-480b-bd77-622e4dfbf59e" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:27.896976 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.896959 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-bound-sa-token" pod "bd9fc616-08fa-480b-bd77-622e4dfbf59e" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:27.901049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.900985 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9fc616-08fa-480b-bd77-622e4dfbf59e-ca-trust-extracted" pod "bd9fc616-08fa-480b-bd77-622e4dfbf59e" (UID: "bd9fc616-08fa-480b-bd77-622e4dfbf59e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:27.995330 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.995296 4968 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-certificates\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:27.995330 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.995325 4968 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-installation-pull-secrets\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:27.995511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.995339 4968 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd9fc616-08fa-480b-bd77-622e4dfbf59e-trusted-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:27.995511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.995352 4968 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd9fc616-08fa-480b-bd77-622e4dfbf59e-ca-trust-extracted\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:27.995511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.995364 4968 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-bound-sa-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:27.995511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.995389 4968 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd9fc616-08fa-480b-bd77-622e4dfbf59e-image-registry-private-configuration\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:27.995511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.995402 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxk4p\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-kube-api-access-hxk4p\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:27.995511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:27.995414 4968 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd9fc616-08fa-480b-bd77-622e4dfbf59e-registry-tls\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:28.115605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:28.115570 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8fff9cc46-btf9d"] Oct 30 14:07:28.125437 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:28.125410 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8fff9cc46-btf9d"] Oct 30 14:07:28.516608 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:28.516533 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9fc616-08fa-480b-bd77-622e4dfbf59e" path="/var/lib/kubelet/pods/bd9fc616-08fa-480b-bd77-622e4dfbf59e/volumes" Oct 30 14:07:49.810403 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:49.810341 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" podUID="fd07a434-a5f6-4cda-a93d-8824978aa590" containerName="registry" containerID="cri-o://0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203" gracePeriod=30 Oct 30 14:07:50.054848 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.054824 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:07:50.178643 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.178615 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-kube-api-access-jv27n\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-kube-api-access-jv27n\") pod \"fd07a434-a5f6-4cda-a93d-8824978aa590\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " Oct 30 14:07:50.178755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.178662 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-trusted-ca\") pod \"fd07a434-a5f6-4cda-a93d-8824978aa590\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " Oct 30 14:07:50.178755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.178702 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-image-registry-private-configuration\") pod \"fd07a434-a5f6-4cda-a93d-8824978aa590\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " Oct 30 14:07:50.178826 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.178789 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-installation-pull-secrets\") pod \"fd07a434-a5f6-4cda-a93d-8824978aa590\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " Oct 30 14:07:50.178826 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.178813 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-certificates\") pod \"fd07a434-a5f6-4cda-a93d-8824978aa590\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " Oct 30 14:07:50.178900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.178845 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") pod \"fd07a434-a5f6-4cda-a93d-8824978aa590\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " Oct 30 14:07:50.179066 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.179041 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-bound-sa-token\") pod \"fd07a434-a5f6-4cda-a93d-8824978aa590\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " Oct 30 14:07:50.179136 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.179107 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd07a434-a5f6-4cda-a93d-8824978aa590-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd07a434-a5f6-4cda-a93d-8824978aa590-ca-trust-extracted\") pod \"fd07a434-a5f6-4cda-a93d-8824978aa590\" (UID: \"fd07a434-a5f6-4cda-a93d-8824978aa590\") " Oct 30 14:07:50.179136 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.179110 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-trusted-ca" pod "fd07a434-a5f6-4cda-a93d-8824978aa590" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:50.179318 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.179287 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-certificates" pod "fd07a434-a5f6-4cda-a93d-8824978aa590" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:50.179499 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.179389 4968 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-certificates\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:50.179499 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.179411 4968 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd07a434-a5f6-4cda-a93d-8824978aa590-trusted-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:50.186426 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.186397 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd07a434-a5f6-4cda-a93d-8824978aa590-ca-trust-extracted" pod "fd07a434-a5f6-4cda-a93d-8824978aa590" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:07:50.201000 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.200975 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls" pod "fd07a434-a5f6-4cda-a93d-8824978aa590" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:50.212780 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.212759 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-bound-sa-token" pod "fd07a434-a5f6-4cda-a93d-8824978aa590" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:50.212780 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.212770 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-kube-api-access-jv27n" pod "fd07a434-a5f6-4cda-a93d-8824978aa590" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590"). InnerVolumeSpecName "kube-api-access-jv27n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:50.212882 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.212773 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-image-registry-private-configuration" pod "fd07a434-a5f6-4cda-a93d-8824978aa590" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:50.212882 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.212759 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-installation-pull-secrets" pod "fd07a434-a5f6-4cda-a93d-8824978aa590" (UID: "fd07a434-a5f6-4cda-a93d-8824978aa590"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:50.280517 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.280489 4968 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-installation-pull-secrets\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:50.280517 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.280517 4968 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-registry-tls\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:50.280682 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.280532 4968 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-bound-sa-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:50.280682 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.280544 4968 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd07a434-a5f6-4cda-a93d-8824978aa590-ca-trust-extracted\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:50.280682 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.280557 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv27n\" (UniqueName: \"kubernetes.io/projected/fd07a434-a5f6-4cda-a93d-8824978aa590-kube-api-access-jv27n\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:50.280682 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.280571 4968 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd07a434-a5f6-4cda-a93d-8824978aa590-image-registry-private-configuration\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:07:50.442499 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:07:50.442410 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:07:50.863506 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.863473 4968 generic.go:356] "Generic (PLEG): container finished" podID="42511697-bcd4-4df3-819c-ab38ba5159c3" containerID="3d6462a55e3ea8c88897e824672873916f2dc2e4a50c2e9358c7e98922c92f8c" exitCode=0 Oct 30 14:07:50.863940 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.863547 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" event={"ID":"42511697-bcd4-4df3-819c-ab38ba5159c3","Type":"ContainerDied","Data":"3d6462a55e3ea8c88897e824672873916f2dc2e4a50c2e9358c7e98922c92f8c"} Oct 30 14:07:50.863940 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.863929 4968 scope.go:122] "RemoveContainer" containerID="3d6462a55e3ea8c88897e824672873916f2dc2e4a50c2e9358c7e98922c92f8c" Oct 30 14:07:50.864858 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.864831 4968 generic.go:356] "Generic (PLEG): container finished" podID="fd07a434-a5f6-4cda-a93d-8824978aa590" containerID="0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203" exitCode=0 Oct 30 14:07:50.864960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.864865 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" event={"ID":"fd07a434-a5f6-4cda-a93d-8824978aa590","Type":"ContainerDied","Data":"0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203"} Oct 30 14:07:50.864960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.864891 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" Oct 30 14:07:50.864960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.864913 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-89594b78c-qxvnh" event={"ID":"fd07a434-a5f6-4cda-a93d-8824978aa590","Type":"ContainerDied","Data":"41c742342fa919c34077244f1e26e10c7bee69f9f6eea40a14950048fe1bd975"} Oct 30 14:07:50.864960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.864939 4968 scope.go:122] "RemoveContainer" containerID="0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203" Oct 30 14:07:50.873481 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.873462 4968 scope.go:122] "RemoveContainer" containerID="0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203" Oct 30 14:07:50.873764 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:07:50.873744 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203\": container with ID starting with 0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203 not found: ID does not exist" containerID="0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203" Oct 30 14:07:50.873839 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.873771 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203"} err="failed to get container status \"0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203\": rpc error: code = NotFound desc = could not find container \"0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203\": container with ID starting with 0270d5b47ef13f5b00e91471204d38e9c5e63c1cb61f365b6108769eed5de203 not found: ID does not exist" Oct 30 14:07:50.931388 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.931367 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-89594b78c-qxvnh"] Oct 30 14:07:50.941524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:50.941498 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-89594b78c-qxvnh"] Oct 30 14:07:51.871301 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:51.871262 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zvbvz" event={"ID":"42511697-bcd4-4df3-819c-ab38ba5159c3","Type":"ContainerStarted","Data":"611dc547f5e313c4e4ddd7c112280688adc5e4d368615cdc62eca396871b86c2"} Oct 30 14:07:52.516413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:52.516384 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd07a434-a5f6-4cda-a93d-8824978aa590" path="/var/lib/kubelet/pods/fd07a434-a5f6-4cda-a93d-8824978aa590/volumes" Oct 30 14:07:59.901933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:59.901898 4968 generic.go:356] "Generic (PLEG): container finished" podID="4171ff7a-5894-49d1-8683-606723fccdf9" containerID="8b38598b9ca7000950dca8428e815e2adbf7d5911fb9245af8e1ea092224ceb1" exitCode=0 Oct 30 14:07:59.902319 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:59.901973 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tfvs9" event={"ID":"4171ff7a-5894-49d1-8683-606723fccdf9","Type":"ContainerDied","Data":"8b38598b9ca7000950dca8428e815e2adbf7d5911fb9245af8e1ea092224ceb1"} Oct 30 14:07:59.902319 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:07:59.902300 4968 scope.go:122] "RemoveContainer" containerID="8b38598b9ca7000950dca8428e815e2adbf7d5911fb9245af8e1ea092224ceb1" Oct 30 14:08:00.642674 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.642638 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-54n2b-bk9ql"] Oct 30 14:08:00.643020 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.643001 4968 cpu_manager.go:414] "containerMap: removing container" podUID="fd07a434-a5f6-4cda-a93d-8824978aa590" containerName="registry" Oct 30 14:08:00.643119 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.643022 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd07a434-a5f6-4cda-a93d-8824978aa590" containerName="registry" Oct 30 14:08:00.643119 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.643046 4968 cpu_manager.go:414] "containerMap: removing container" podUID="bd9fc616-08fa-480b-bd77-622e4dfbf59e" containerName="registry" Oct 30 14:08:00.643119 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.643055 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9fc616-08fa-480b-bd77-622e4dfbf59e" containerName="registry" Oct 30 14:08:00.643222 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.643133 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9fc616-08fa-480b-bd77-622e4dfbf59e" containerName="registry" Oct 30 14:08:00.643222 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.643147 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd07a434-a5f6-4cda-a93d-8824978aa590" containerName="registry" Oct 30 14:08:00.646335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.646313 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.668572 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.668550 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h2cvg\"" Oct 30 14:08:00.669447 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.669428 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-54n2b-bk9ql"] Oct 30 14:08:00.764650 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.764618 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-service-ca-bundle\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.764833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.764660 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c217ef62-765a-491f-9533-3f43b20d38cb-archives-path\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.764833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.764680 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-trusted-ca-bundle\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.764910 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.764832 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57zwb\" (UniqueName: \"kubernetes.io/projected/c217ef62-765a-491f-9533-3f43b20d38cb-kube-api-access-57zwb\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.866084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.866045 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-service-ca-bundle\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.866256 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.866093 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c217ef62-765a-491f-9533-3f43b20d38cb-archives-path\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.866256 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.866118 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-trusted-ca-bundle\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.866256 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.866164 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57zwb\" (UniqueName: \"kubernetes.io/projected/c217ef62-765a-491f-9533-3f43b20d38cb-kube-api-access-57zwb\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.866612 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.866588 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c217ef62-765a-491f-9533-3f43b20d38cb-archives-path\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.866822 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.866801 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-service-ca-bundle\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.866934 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.866915 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-trusted-ca-bundle\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.888304 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.888271 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57zwb\" (UniqueName: \"kubernetes.io/projected/c217ef62-765a-491f-9533-3f43b20d38cb-kube-api-access-57zwb\") pod \"periodic-gathering-54n2b-bk9ql\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:00.907834 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.907760 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tfvs9" event={"ID":"4171ff7a-5894-49d1-8683-606723fccdf9","Type":"ContainerStarted","Data":"a5741c2be9b9f27b19b88c0e3a9b2814dac551cf75f926b194292909d10b4cfa"} Oct 30 14:08:00.957605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:00.957576 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:08:01.256160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:01.256084 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-54n2b-bk9ql"] Oct 30 14:08:01.260702 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:08:01.260675 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc217ef62_765a_491f_9533_3f43b20d38cb.slice/crio-2a48402c2ebf051d465957f0b0ee1fc8c6da49304fac7041947c91f9e8cb6f79 WatchSource:0}: Error finding container 2a48402c2ebf051d465957f0b0ee1fc8c6da49304fac7041947c91f9e8cb6f79: Status 404 returned error can't find the container with id 2a48402c2ebf051d465957f0b0ee1fc8c6da49304fac7041947c91f9e8cb6f79 Oct 30 14:08:01.913617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:01.913574 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54n2b-bk9ql" event={"ID":"c217ef62-765a-491f-9533-3f43b20d38cb","Type":"ContainerStarted","Data":"37904dcda31ffc9c9495089eaa86d29c9f1e51bd38d91ba2e7ae96313f5c8c71"} Oct 30 14:08:01.914022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:01.913627 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54n2b-bk9ql" event={"ID":"c217ef62-765a-491f-9533-3f43b20d38cb","Type":"ContainerStarted","Data":"2a48402c2ebf051d465957f0b0ee1fc8c6da49304fac7041947c91f9e8cb6f79"} Oct 30 14:08:01.950206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:01.950153 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-54n2b-bk9ql" podStartSLOduration=1.9501357449999999 podStartE2EDuration="1.950135745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:01.949191301 +0000 UTC m=+77.574634675" watchObservedRunningTime="2026-10-30 14:08:01.950135745 +0000 UTC m=+77.575579117" Oct 30 14:08:10.948196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:10.948161 4968 generic.go:356] "Generic (PLEG): container finished" podID="fb9d3ab7-8a7f-4ccd-a336-912b00332110" containerID="fa73e14891af6c9f8196a20bcfd38911d9a5116d5269f6a7a47df2b46184dce2" exitCode=0 Oct 30 14:08:10.948584 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:10.948222 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" event={"ID":"fb9d3ab7-8a7f-4ccd-a336-912b00332110","Type":"ContainerDied","Data":"fa73e14891af6c9f8196a20bcfd38911d9a5116d5269f6a7a47df2b46184dce2"} Oct 30 14:08:10.948584 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:10.948531 4968 scope.go:122] "RemoveContainer" containerID="fa73e14891af6c9f8196a20bcfd38911d9a5116d5269f6a7a47df2b46184dce2" Oct 30 14:08:11.954206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:08:11.954172 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-892zh" event={"ID":"fb9d3ab7-8a7f-4ccd-a336-912b00332110","Type":"ContainerStarted","Data":"a6de2da99a646dfcc64fa185a355dcfd1c00e52f5f5f24e267a31e9e09cd77c0"} Oct 30 14:08:20.448073 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:08:20.448031 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:08:50.441982 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:08:50.441940 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:09:20.442610 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:09:20.442569 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:09:26.231224 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:26.231188 4968 generic.go:356] "Generic (PLEG): container finished" podID="c217ef62-765a-491f-9533-3f43b20d38cb" containerID="37904dcda31ffc9c9495089eaa86d29c9f1e51bd38d91ba2e7ae96313f5c8c71" exitCode=0 Oct 30 14:09:26.231607 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:26.231258 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54n2b-bk9ql" event={"ID":"c217ef62-765a-491f-9533-3f43b20d38cb","Type":"ContainerDied","Data":"37904dcda31ffc9c9495089eaa86d29c9f1e51bd38d91ba2e7ae96313f5c8c71"} Oct 30 14:09:27.356088 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.356067 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:09:27.516190 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.516110 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-service-ca-bundle\") pod \"c217ef62-765a-491f-9533-3f43b20d38cb\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " Oct 30 14:09:27.516190 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.516156 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c217ef62-765a-491f-9533-3f43b20d38cb-kube-api-access-57zwb\" (UniqueName: \"kubernetes.io/projected/c217ef62-765a-491f-9533-3f43b20d38cb-kube-api-access-57zwb\") pod \"c217ef62-765a-491f-9533-3f43b20d38cb\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " Oct 30 14:09:27.516190 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.516187 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-trusted-ca-bundle\") pod \"c217ef62-765a-491f-9533-3f43b20d38cb\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " Oct 30 14:09:27.516448 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.516244 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c217ef62-765a-491f-9533-3f43b20d38cb-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c217ef62-765a-491f-9533-3f43b20d38cb-archives-path\") pod \"c217ef62-765a-491f-9533-3f43b20d38cb\" (UID: \"c217ef62-765a-491f-9533-3f43b20d38cb\") " Oct 30 14:09:27.516544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.516519 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-service-ca-bundle" pod "c217ef62-765a-491f-9533-3f43b20d38cb" (UID: "c217ef62-765a-491f-9533-3f43b20d38cb"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:09:27.516604 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.516570 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c217ef62-765a-491f-9533-3f43b20d38cb-archives-path" pod "c217ef62-765a-491f-9533-3f43b20d38cb" (UID: "c217ef62-765a-491f-9533-3f43b20d38cb"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:09:27.516604 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.516577 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-trusted-ca-bundle" pod "c217ef62-765a-491f-9533-3f43b20d38cb" (UID: "c217ef62-765a-491f-9533-3f43b20d38cb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:09:27.518503 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.518481 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c217ef62-765a-491f-9533-3f43b20d38cb-kube-api-access-57zwb" pod "c217ef62-765a-491f-9533-3f43b20d38cb" (UID: "c217ef62-765a-491f-9533-3f43b20d38cb"). InnerVolumeSpecName "kube-api-access-57zwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:09:27.616763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.616731 4968 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-trusted-ca-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:09:27.616763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.616758 4968 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c217ef62-765a-491f-9533-3f43b20d38cb-archives-path\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:09:27.616763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.616768 4968 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c217ef62-765a-491f-9533-3f43b20d38cb-service-ca-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:09:27.616953 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:27.616777 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57zwb\" (UniqueName: \"kubernetes.io/projected/c217ef62-765a-491f-9533-3f43b20d38cb-kube-api-access-57zwb\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:09:28.240943 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:28.240913 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54n2b-bk9ql" Oct 30 14:09:28.241106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:28.240916 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54n2b-bk9ql" event={"ID":"c217ef62-765a-491f-9533-3f43b20d38cb","Type":"ContainerDied","Data":"2a48402c2ebf051d465957f0b0ee1fc8c6da49304fac7041947c91f9e8cb6f79"} Oct 30 14:09:28.241106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:09:28.241033 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a48402c2ebf051d465957f0b0ee1fc8c6da49304fac7041947c91f9e8cb6f79" Oct 30 14:09:50.442524 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:09:50.442480 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:10:20.443575 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:10:20.443534 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:10:50.443261 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:10:50.443220 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:11:20.442689 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:11:20.442654 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:11:44.530293 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:44.530266 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:11:44.538263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:44.538235 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:11:50.443025 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:11:50.442980 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:11:58.941297 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:58.941217 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-942cn"] Oct 30 14:11:58.941675 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:58.941550 4968 cpu_manager.go:414] "containerMap: removing container" podUID="c217ef62-765a-491f-9533-3f43b20d38cb" containerName="insights-gathering" Oct 30 14:11:58.941675 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:58.941565 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c217ef62-765a-491f-9533-3f43b20d38cb" containerName="insights-gathering" Oct 30 14:11:58.941675 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:58.941630 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="c217ef62-765a-491f-9533-3f43b20d38cb" containerName="insights-gathering" Oct 30 14:11:58.944446 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:58.944425 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-942cn" Oct 30 14:11:58.957146 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:58.957124 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 14:11:58.966525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:58.966499 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 14:11:58.966647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:58.966598 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-972rd\"" Oct 30 14:11:58.967644 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:58.967619 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-942cn"] Oct 30 14:11:59.018560 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:59.018523 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5fh\" (UniqueName: \"kubernetes.io/projected/a88f21b0-a6dd-4646-ac34-a404b869a66f-kube-api-access-bx5fh\") pod \"cert-manager-597b95cb84-942cn\" (UID: \"a88f21b0-a6dd-4646-ac34-a404b869a66f\") " pod="cert-manager/cert-manager-597b95cb84-942cn" Oct 30 14:11:59.018737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:59.018635 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a88f21b0-a6dd-4646-ac34-a404b869a66f-bound-sa-token\") pod \"cert-manager-597b95cb84-942cn\" (UID: \"a88f21b0-a6dd-4646-ac34-a404b869a66f\") " pod="cert-manager/cert-manager-597b95cb84-942cn" Oct 30 14:11:59.119796 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:59.119765 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a88f21b0-a6dd-4646-ac34-a404b869a66f-bound-sa-token\") pod \"cert-manager-597b95cb84-942cn\" (UID: \"a88f21b0-a6dd-4646-ac34-a404b869a66f\") " pod="cert-manager/cert-manager-597b95cb84-942cn" Oct 30 14:11:59.119943 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:59.119808 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5fh\" (UniqueName: \"kubernetes.io/projected/a88f21b0-a6dd-4646-ac34-a404b869a66f-kube-api-access-bx5fh\") pod \"cert-manager-597b95cb84-942cn\" (UID: \"a88f21b0-a6dd-4646-ac34-a404b869a66f\") " pod="cert-manager/cert-manager-597b95cb84-942cn" Oct 30 14:11:59.141904 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:59.141882 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a88f21b0-a6dd-4646-ac34-a404b869a66f-bound-sa-token\") pod \"cert-manager-597b95cb84-942cn\" (UID: \"a88f21b0-a6dd-4646-ac34-a404b869a66f\") " pod="cert-manager/cert-manager-597b95cb84-942cn" Oct 30 14:11:59.142037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:59.142019 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5fh\" (UniqueName: \"kubernetes.io/projected/a88f21b0-a6dd-4646-ac34-a404b869a66f-kube-api-access-bx5fh\") pod \"cert-manager-597b95cb84-942cn\" (UID: \"a88f21b0-a6dd-4646-ac34-a404b869a66f\") " pod="cert-manager/cert-manager-597b95cb84-942cn" Oct 30 14:11:59.264145 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:59.264065 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-942cn" Oct 30 14:11:59.562050 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:59.561975 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-942cn"] Oct 30 14:11:59.566636 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:11:59.566609 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda88f21b0_a6dd_4646_ac34_a404b869a66f.slice/crio-b15f1f05f6fd27f48ca49b10f66fe0e7ed6f342a985bc13a9ae85337aa3e0ec1 WatchSource:0}: Error finding container b15f1f05f6fd27f48ca49b10f66fe0e7ed6f342a985bc13a9ae85337aa3e0ec1: Status 404 returned error can't find the container with id b15f1f05f6fd27f48ca49b10f66fe0e7ed6f342a985bc13a9ae85337aa3e0ec1 Oct 30 14:11:59.568607 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:59.568590 4968 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:11:59.799173 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:11:59.799135 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-942cn" event={"ID":"a88f21b0-a6dd-4646-ac34-a404b869a66f","Type":"ContainerStarted","Data":"b15f1f05f6fd27f48ca49b10f66fe0e7ed6f342a985bc13a9ae85337aa3e0ec1"} Oct 30 14:12:01.544925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.544888 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-p4f65"] Oct 30 14:12:01.553017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.552988 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-p4f65" Oct 30 14:12:01.578042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.578015 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 14:12:01.578224 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.578092 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kc758\"" Oct 30 14:12:01.578621 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.578595 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:12:01.579214 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.579195 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-p4f65"] Oct 30 14:12:01.641850 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.641810 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0093dfb6-dba7-4dee-a699-37a8c5a946d5-tmp\") pod \"openshift-lws-operator-8465994787-p4f65\" (UID: \"0093dfb6-dba7-4dee-a699-37a8c5a946d5\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-p4f65" Oct 30 14:12:01.642049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.641885 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj6zb\" (UniqueName: \"kubernetes.io/projected/0093dfb6-dba7-4dee-a699-37a8c5a946d5-kube-api-access-lj6zb\") pod \"openshift-lws-operator-8465994787-p4f65\" (UID: \"0093dfb6-dba7-4dee-a699-37a8c5a946d5\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-p4f65" Oct 30 14:12:01.743075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.743039 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6zb\" (UniqueName: \"kubernetes.io/projected/0093dfb6-dba7-4dee-a699-37a8c5a946d5-kube-api-access-lj6zb\") pod \"openshift-lws-operator-8465994787-p4f65\" (UID: \"0093dfb6-dba7-4dee-a699-37a8c5a946d5\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-p4f65" Oct 30 14:12:01.743259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.743128 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0093dfb6-dba7-4dee-a699-37a8c5a946d5-tmp\") pod \"openshift-lws-operator-8465994787-p4f65\" (UID: \"0093dfb6-dba7-4dee-a699-37a8c5a946d5\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-p4f65" Oct 30 14:12:01.743531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.743511 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0093dfb6-dba7-4dee-a699-37a8c5a946d5-tmp\") pod \"openshift-lws-operator-8465994787-p4f65\" (UID: \"0093dfb6-dba7-4dee-a699-37a8c5a946d5\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-p4f65" Oct 30 14:12:01.765883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.765851 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj6zb\" (UniqueName: \"kubernetes.io/projected/0093dfb6-dba7-4dee-a699-37a8c5a946d5-kube-api-access-lj6zb\") pod \"openshift-lws-operator-8465994787-p4f65\" (UID: \"0093dfb6-dba7-4dee-a699-37a8c5a946d5\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-p4f65" Oct 30 14:12:01.866678 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:01.866649 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-p4f65" Oct 30 14:12:02.169002 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:02.168930 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-p4f65"] Oct 30 14:12:02.174779 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:12:02.174747 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0093dfb6_dba7_4dee_a699_37a8c5a946d5.slice/crio-8f2d30a2a410c64fa96747b35cfd496e1a99a907ebcb9b2696e2550d7f341d90 WatchSource:0}: Error finding container 8f2d30a2a410c64fa96747b35cfd496e1a99a907ebcb9b2696e2550d7f341d90: Status 404 returned error can't find the container with id 8f2d30a2a410c64fa96747b35cfd496e1a99a907ebcb9b2696e2550d7f341d90 Oct 30 14:12:02.813419 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:02.813289 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-p4f65" event={"ID":"0093dfb6-dba7-4dee-a699-37a8c5a946d5","Type":"ContainerStarted","Data":"8f2d30a2a410c64fa96747b35cfd496e1a99a907ebcb9b2696e2550d7f341d90"} Oct 30 14:12:02.815248 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:02.815210 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-942cn" event={"ID":"a88f21b0-a6dd-4646-ac34-a404b869a66f","Type":"ContainerStarted","Data":"9939b5fd887a03459deb29a863723a9a30e5aad124f71769c89896c30582b03e"} Oct 30 14:12:02.855870 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:02.855817 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-942cn" podStartSLOduration=1.878232162 podStartE2EDuration="4.855799533s" totalImagesPullingTime="2.977567371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:02.854803714 +0000 UTC m=+318.480247089" watchObservedRunningTime="2026-10-30 14:12:02.855799533 +0000 UTC m=+318.481242908" Oct 30 14:12:04.824206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:04.824127 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-p4f65" event={"ID":"0093dfb6-dba7-4dee-a699-37a8c5a946d5","Type":"ContainerStarted","Data":"40c430df25f6ae015732dd20a9c6fe147840cdcf3f3cc3e6ae851f2c0bf583f9"} Oct 30 14:12:04.862395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:04.862343 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-p4f65" podStartSLOduration=1.519813954 podStartE2EDuration="3.862328514s" totalImagesPullingTime="2.34251456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:04.862275816 +0000 UTC m=+320.487719190" watchObservedRunningTime="2026-10-30 14:12:04.862328514 +0000 UTC m=+320.487771884" Oct 30 14:12:20.442822 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:12:20.442784 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:12:23.425340 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.425298 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr"] Oct 30 14:12:23.429108 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.429087 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.466267 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.466240 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr"] Oct 30 14:12:23.508762 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.508726 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 14:12:23.508903 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.508781 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 14:12:23.508903 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.508864 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6f7bn\"" Oct 30 14:12:23.509014 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.508906 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 14:12:23.509014 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.508941 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 14:12:23.509103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.509034 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 14:12:23.515908 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.515887 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a7bf92ce-2f0a-474c-b69a-a568080a65aa-manifests\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.515996 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.515916 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a7bf92ce-2f0a-474c-b69a-a568080a65aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.516033 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.516008 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7bf92ce-2f0a-474c-b69a-a568080a65aa-tmp\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.516066 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.516051 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7bf92ce-2f0a-474c-b69a-a568080a65aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.516099 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.516073 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55xz\" (UniqueName: \"kubernetes.io/projected/a7bf92ce-2f0a-474c-b69a-a568080a65aa-kube-api-access-v55xz\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.516148 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.516136 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7bf92ce-2f0a-474c-b69a-a568080a65aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.617470 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.617439 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7bf92ce-2f0a-474c-b69a-a568080a65aa-tmp\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.617617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.617501 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7bf92ce-2f0a-474c-b69a-a568080a65aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.617617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.617532 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55xz\" (UniqueName: \"kubernetes.io/projected/a7bf92ce-2f0a-474c-b69a-a568080a65aa-kube-api-access-v55xz\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.617617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.617571 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7bf92ce-2f0a-474c-b69a-a568080a65aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.617617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.617602 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a7bf92ce-2f0a-474c-b69a-a568080a65aa-manifests\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.617871 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.617640 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a7bf92ce-2f0a-474c-b69a-a568080a65aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.618057 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.618029 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a7bf92ce-2f0a-474c-b69a-a568080a65aa-manifests\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.619753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.619730 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7bf92ce-2f0a-474c-b69a-a568080a65aa-tmp\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.620095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.620074 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a7bf92ce-2f0a-474c-b69a-a568080a65aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.620172 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.620075 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7bf92ce-2f0a-474c-b69a-a568080a65aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.620172 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.620148 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7bf92ce-2f0a-474c-b69a-a568080a65aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.640084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.640061 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55xz\" (UniqueName: \"kubernetes.io/projected/a7bf92ce-2f0a-474c-b69a-a568080a65aa-kube-api-access-v55xz\") pod \"opendatahub-operator-controller-manager-6749b785b4-84kgr\" (UID: \"a7bf92ce-2f0a-474c-b69a-a568080a65aa\") " pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:23.743915 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:23.743834 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:24.101578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:24.101548 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr"] Oct 30 14:12:24.105124 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:12:24.105092 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7bf92ce_2f0a_474c_b69a_a568080a65aa.slice/crio-710617e7ff18c13168bc8e53ad8ee9131fc5cec97d16a7b36103db1265de6d8c WatchSource:0}: Error finding container 710617e7ff18c13168bc8e53ad8ee9131fc5cec97d16a7b36103db1265de6d8c: Status 404 returned error can't find the container with id 710617e7ff18c13168bc8e53ad8ee9131fc5cec97d16a7b36103db1265de6d8c Oct 30 14:12:24.907818 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:24.907777 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" event={"ID":"a7bf92ce-2f0a-474c-b69a-a568080a65aa","Type":"ContainerStarted","Data":"710617e7ff18c13168bc8e53ad8ee9131fc5cec97d16a7b36103db1265de6d8c"} Oct 30 14:12:27.921485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:27.921445 4968 generic.go:356] "Generic (PLEG): container finished" podID="a7bf92ce-2f0a-474c-b69a-a568080a65aa" containerID="edd8dd212560e21e50b641506f91d42090bb53d4925c0f00a91c6fde74bb65c2" exitCode=0 Oct 30 14:12:27.921915 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:27.921492 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" event={"ID":"a7bf92ce-2f0a-474c-b69a-a568080a65aa","Type":"ContainerDied","Data":"edd8dd212560e21e50b641506f91d42090bb53d4925c0f00a91c6fde74bb65c2"} Oct 30 14:12:28.927062 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:28.927026 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" event={"ID":"a7bf92ce-2f0a-474c-b69a-a568080a65aa","Type":"ContainerStarted","Data":"0f4c6b1f371620c67dac98521b32ca4fc2d2a798c7a9756afee026d912ef2fec"} Oct 30 14:12:28.927539 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:28.927120 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:29.015041 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:29.014991 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" podStartSLOduration=2.262716721 podStartE2EDuration="6.014977062s" totalImagesPullingTime="3.269417984s" totalInitContainerRuntime="482.842357ms" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:29.003678473 +0000 UTC m=+344.629121847" watchObservedRunningTime="2026-10-30 14:12:29.014977062 +0000 UTC m=+344.640420435" Oct 30 14:12:39.933041 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:39.933009 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6749b785b4-84kgr" Oct 30 14:12:43.125278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.125244 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx"] Oct 30 14:12:43.135299 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.135276 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" Oct 30 14:12:43.160087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.160058 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 14:12:43.160219 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.160084 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7rf2q\"" Oct 30 14:12:43.160219 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.160092 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 14:12:43.161336 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.161317 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx"] Oct 30 14:12:43.175498 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.175477 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f369d28-d4eb-4d7c-81b6-b80cd88edb80-tls-certs\") pod \"kube-auth-proxy-847b5f84b8-7m6sx\" (UID: \"9f369d28-d4eb-4d7c-81b6-b80cd88edb80\") " pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" Oct 30 14:12:43.175611 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.175526 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q927\" (UniqueName: \"kubernetes.io/projected/9f369d28-d4eb-4d7c-81b6-b80cd88edb80-kube-api-access-7q927\") pod \"kube-auth-proxy-847b5f84b8-7m6sx\" (UID: \"9f369d28-d4eb-4d7c-81b6-b80cd88edb80\") " pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" Oct 30 14:12:43.175852 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.175647 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f369d28-d4eb-4d7c-81b6-b80cd88edb80-tmp\") pod \"kube-auth-proxy-847b5f84b8-7m6sx\" (UID: \"9f369d28-d4eb-4d7c-81b6-b80cd88edb80\") " pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" Oct 30 14:12:43.276180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.276146 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f369d28-d4eb-4d7c-81b6-b80cd88edb80-tls-certs\") pod \"kube-auth-proxy-847b5f84b8-7m6sx\" (UID: \"9f369d28-d4eb-4d7c-81b6-b80cd88edb80\") " pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" Oct 30 14:12:43.276349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.276213 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q927\" (UniqueName: \"kubernetes.io/projected/9f369d28-d4eb-4d7c-81b6-b80cd88edb80-kube-api-access-7q927\") pod \"kube-auth-proxy-847b5f84b8-7m6sx\" (UID: \"9f369d28-d4eb-4d7c-81b6-b80cd88edb80\") " pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" Oct 30 14:12:43.276349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.276249 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f369d28-d4eb-4d7c-81b6-b80cd88edb80-tmp\") pod \"kube-auth-proxy-847b5f84b8-7m6sx\" (UID: \"9f369d28-d4eb-4d7c-81b6-b80cd88edb80\") " pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" Oct 30 14:12:43.278626 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.278598 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f369d28-d4eb-4d7c-81b6-b80cd88edb80-tmp\") pod \"kube-auth-proxy-847b5f84b8-7m6sx\" (UID: \"9f369d28-d4eb-4d7c-81b6-b80cd88edb80\") " pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" Oct 30 14:12:43.278890 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.278867 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f369d28-d4eb-4d7c-81b6-b80cd88edb80-tls-certs\") pod \"kube-auth-proxy-847b5f84b8-7m6sx\" (UID: \"9f369d28-d4eb-4d7c-81b6-b80cd88edb80\") " pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" Oct 30 14:12:43.297886 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.297859 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q927\" (UniqueName: \"kubernetes.io/projected/9f369d28-d4eb-4d7c-81b6-b80cd88edb80-kube-api-access-7q927\") pod \"kube-auth-proxy-847b5f84b8-7m6sx\" (UID: \"9f369d28-d4eb-4d7c-81b6-b80cd88edb80\") " pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" Oct 30 14:12:43.447272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.447191 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" Oct 30 14:12:43.746062 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.745968 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx"] Oct 30 14:12:43.750704 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:12:43.750674 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f369d28_d4eb_4d7c_81b6_b80cd88edb80.slice/crio-99356b20b86e9900ee1a258ee20c50406e79c462c9d57a6351520ee318b73aad WatchSource:0}: Error finding container 99356b20b86e9900ee1a258ee20c50406e79c462c9d57a6351520ee318b73aad: Status 404 returned error can't find the container with id 99356b20b86e9900ee1a258ee20c50406e79c462c9d57a6351520ee318b73aad Oct 30 14:12:43.991526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:43.991490 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" event={"ID":"9f369d28-d4eb-4d7c-81b6-b80cd88edb80","Type":"ContainerStarted","Data":"99356b20b86e9900ee1a258ee20c50406e79c462c9d57a6351520ee318b73aad"} Oct 30 14:12:44.768864 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.768832 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2"] Oct 30 14:12:44.773986 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.773955 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.788745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.788683 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 14:12:44.789016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.788996 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 14:12:44.789094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.789034 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 14:12:44.790390 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.790365 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn6qv\" (UniqueName: \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-kube-api-access-mn6qv\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.790491 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.790420 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.790583 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.790533 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/366ae49d-567d-46c9-a21f-07a796f2e34c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.790662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.790594 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.790662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.790614 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.790662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.790638 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.790865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.790844 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.798188 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.798153 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 14:12:44.798188 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.798173 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8s22s\"" Oct 30 14:12:44.800471 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.800436 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2"] Oct 30 14:12:44.891551 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.891518 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/366ae49d-567d-46c9-a21f-07a796f2e34c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.891551 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.891556 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.891789 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.891578 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.891789 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.891755 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.891904 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.891816 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.891957 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.891899 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn6qv\" (UniqueName: \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-kube-api-access-mn6qv\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.892010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.891955 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.892560 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.892526 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.893908 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.893870 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/366ae49d-567d-46c9-a21f-07a796f2e34c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.894050 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.894014 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.894115 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.894087 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.894263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.894247 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.914584 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.914553 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:44.915244 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:44.915223 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn6qv\" (UniqueName: \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-kube-api-access-mn6qv\") pod \"istiod-openshift-gateway-7876bfcb74-96zq2\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:45.089240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:45.089213 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:45.414173 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:45.414043 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2"] Oct 30 14:12:45.421684 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:12:45.421640 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366ae49d_567d_46c9_a21f_07a796f2e34c.slice/crio-f7cdfa1f2bfdeed2fcb712ff750b556e214e078355df07f158e21e7aaa7faccf WatchSource:0}: Error finding container f7cdfa1f2bfdeed2fcb712ff750b556e214e078355df07f158e21e7aaa7faccf: Status 404 returned error can't find the container with id f7cdfa1f2bfdeed2fcb712ff750b556e214e078355df07f158e21e7aaa7faccf Oct 30 14:12:46.004411 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:46.004369 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" event={"ID":"366ae49d-567d-46c9-a21f-07a796f2e34c","Type":"ContainerStarted","Data":"f7cdfa1f2bfdeed2fcb712ff750b556e214e078355df07f158e21e7aaa7faccf"} Oct 30 14:12:46.174473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:46.174443 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 14:12:47.010866 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:47.010822 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" event={"ID":"9f369d28-d4eb-4d7c-81b6-b80cd88edb80","Type":"ContainerStarted","Data":"1f4cb727145db58702ca9f33f639f3b8f3dda63dfcec381bb6d5fe499856df29"} Oct 30 14:12:47.051747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:47.051686 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-847b5f84b8-7m6sx" podStartSLOduration=1.6417244819999999 podStartE2EDuration="4.051644076s" totalImagesPullingTime="2.409919594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:47.050858711 +0000 UTC m=+362.676302084" watchObservedRunningTime="2026-10-30 14:12:47.051644076 +0000 UTC m=+362.677087452" Oct 30 14:12:47.977805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:47.977758 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:12:47.977905 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:47.977835 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:12:49.021396 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:49.021354 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" event={"ID":"366ae49d-567d-46c9-a21f-07a796f2e34c","Type":"ContainerStarted","Data":"24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289"} Oct 30 14:12:49.021822 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:49.021561 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:49.023210 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:49.023182 4968 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-96zq2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:12:49.023346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:49.023237 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" podUID="366ae49d-567d-46c9-a21f-07a796f2e34c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:12:49.086120 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:49.086079 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" podStartSLOduration=2.535799626 podStartE2EDuration="5.086067053s" totalImagesPullingTime="2.550267427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:49.076057121 +0000 UTC m=+364.701500495" watchObservedRunningTime="2026-10-30 14:12:49.086067053 +0000 UTC m=+364.711510426" Oct 30 14:12:50.025472 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:12:50.025437 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:12:50.442998 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:12:50.442965 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:13:01.722865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.722827 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m"] Oct 30 14:13:01.731474 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.731439 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.754991 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.754966 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-wp89t\"" Oct 30 14:13:01.776045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.776013 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 14:13:01.782173 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.781802 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m"] Oct 30 14:13:01.838431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.838391 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/54b9a0e6-1899-40b6-836a-4dae7752ce36-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.838431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.838440 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b9a0e6-1899-40b6-836a-4dae7752ce36-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.838657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.838464 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b9a0e6-1899-40b6-836a-4dae7752ce36-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.838657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.838623 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99pcv\" (UniqueName: \"kubernetes.io/projected/54b9a0e6-1899-40b6-836a-4dae7752ce36-kube-api-access-99pcv\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.939798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.939756 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99pcv\" (UniqueName: \"kubernetes.io/projected/54b9a0e6-1899-40b6-836a-4dae7752ce36-kube-api-access-99pcv\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.939991 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.939816 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/54b9a0e6-1899-40b6-836a-4dae7752ce36-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.939991 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.939846 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b9a0e6-1899-40b6-836a-4dae7752ce36-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.939991 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.939868 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b9a0e6-1899-40b6-836a-4dae7752ce36-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.940302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.940277 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/54b9a0e6-1899-40b6-836a-4dae7752ce36-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.940374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.940335 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b9a0e6-1899-40b6-836a-4dae7752ce36-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.940791 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.940772 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b9a0e6-1899-40b6-836a-4dae7752ce36-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:01.962487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:01.962463 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99pcv\" (UniqueName: \"kubernetes.io/projected/54b9a0e6-1899-40b6-836a-4dae7752ce36-kube-api-access-99pcv\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjg7m\" (UID: \"54b9a0e6-1899-40b6-836a-4dae7752ce36\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:02.044269 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:02.044182 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:02.358005 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:02.357972 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m"] Oct 30 14:13:02.362568 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:13:02.362541 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b9a0e6_1899_40b6_836a_4dae7752ce36.slice/crio-a6d31ac158edca252e033c273827fa46b27e93ef24a533b2a4951315bd2d8023 WatchSource:0}: Error finding container a6d31ac158edca252e033c273827fa46b27e93ef24a533b2a4951315bd2d8023: Status 404 returned error can't find the container with id a6d31ac158edca252e033c273827fa46b27e93ef24a533b2a4951315bd2d8023 Oct 30 14:13:03.082663 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:03.082621 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" event={"ID":"54b9a0e6-1899-40b6-836a-4dae7752ce36","Type":"ContainerStarted","Data":"a6d31ac158edca252e033c273827fa46b27e93ef24a533b2a4951315bd2d8023"} Oct 30 14:13:04.087825 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:04.087790 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" event={"ID":"54b9a0e6-1899-40b6-836a-4dae7752ce36","Type":"ContainerStarted","Data":"0f9d53be62fdd6ec69e20682ed9250ec86ea9d6e3d4e10c5a309f0b5518ab005"} Oct 30 14:13:04.088325 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:04.087867 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:04.139476 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:04.139435 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" podStartSLOduration=1.6080867319999999 podStartE2EDuration="3.139420554s" totalImagesPullingTime="1.531333822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:04.138208363 +0000 UTC m=+379.763651736" watchObservedRunningTime="2026-10-30 14:13:04.139420554 +0000 UTC m=+379.764863931" Oct 30 14:13:20.443241 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:13:20.443206 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:13:25.093457 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:25.093421 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjg7m" Oct 30 14:13:48.022034 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.021997 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk"] Oct 30 14:13:48.031104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.031080 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.050259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.050236 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk"] Oct 30 14:13:48.112244 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.112208 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.112244 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.112246 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.112451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.112337 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.112451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.112368 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.112451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.112400 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.112566 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.112469 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.112566 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.112551 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4w9\" (UniqueName: \"kubernetes.io/projected/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-kube-api-access-dx4w9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.213173 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.213137 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.213173 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.213176 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.213407 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.213201 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.213407 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.213222 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.213407 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.213291 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4w9\" (UniqueName: \"kubernetes.io/projected/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-kube-api-access-dx4w9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.213407 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.213325 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.213625 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.213513 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.214026 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.214002 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.215709 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.215680 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.215859 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.215688 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.215929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.215889 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.216021 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.216001 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.240610 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.240581 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.240705 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.240622 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4w9\" (UniqueName: \"kubernetes.io/projected/13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b-kube-api-access-dx4w9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2cgmk\" (UID: \"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.343050 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.343023 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:48.662411 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.662326 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk"] Oct 30 14:13:48.666664 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:13:48.666632 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b59ffb_ff1e_4a12_987b_dc7f5daf9b9b.slice/crio-736c0d031dfb929b1a78afc1035fbd4b66e2a6f890a5e4d1632145081dff7a7a WatchSource:0}: Error finding container 736c0d031dfb929b1a78afc1035fbd4b66e2a6f890a5e4d1632145081dff7a7a: Status 404 returned error can't find the container with id 736c0d031dfb929b1a78afc1035fbd4b66e2a6f890a5e4d1632145081dff7a7a Oct 30 14:13:48.668800 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.668768 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:13:48.668879 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:48.668840 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:13:49.275804 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:49.275761 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" event={"ID":"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b","Type":"ContainerStarted","Data":"f1bfef965edcfb341a3672598d8bf6d09f6869d9409409816e9c6f501e13247c"} Oct 30 14:13:49.276206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:49.275818 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" event={"ID":"13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b","Type":"ContainerStarted","Data":"736c0d031dfb929b1a78afc1035fbd4b66e2a6f890a5e4d1632145081dff7a7a"} Oct 30 14:13:49.276206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:49.275934 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:49.277478 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:49.277448 4968 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2cgmk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:13:49.277587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:49.277506 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" podUID="13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:13:49.320428 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:49.320376 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" podStartSLOduration=2.320361521 podStartE2EDuration="2.320361521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:49.319438052 +0000 UTC m=+424.944881425" watchObservedRunningTime="2026-10-30 14:13:49.320361521 +0000 UTC m=+424.945804893" Oct 30 14:13:50.280964 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.280933 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2cgmk" Oct 30 14:13:50.370507 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.370470 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2"] Oct 30 14:13:50.371317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.370789 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" podUID="366ae49d-567d-46c9-a21f-07a796f2e34c" containerName="discovery" containerID="cri-o://24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289" gracePeriod=30 Oct 30 14:13:50.444147 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:13:50.444115 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:13:50.613073 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.613047 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:13:50.634024 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.633993 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-kube-api-access-mn6qv\" (UniqueName: \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-kube-api-access-mn6qv\") pod \"366ae49d-567d-46c9-a21f-07a796f2e34c\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " Oct 30 14:13:50.634160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.634039 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-ca-configmap\") pod \"366ae49d-567d-46c9-a21f-07a796f2e34c\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " Oct 30 14:13:50.634160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.634073 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-cacerts\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-cacerts\") pod \"366ae49d-567d-46c9-a21f-07a796f2e34c\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " Oct 30 14:13:50.634160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.634092 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-dns-cert\") pod \"366ae49d-567d-46c9-a21f-07a796f2e34c\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " Oct 30 14:13:50.634160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.634143 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-kubeconfig\") pod \"366ae49d-567d-46c9-a21f-07a796f2e34c\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " Oct 30 14:13:50.634393 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.634181 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/366ae49d-567d-46c9-a21f-07a796f2e34c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/366ae49d-567d-46c9-a21f-07a796f2e34c-local-certs\") pod \"366ae49d-567d-46c9-a21f-07a796f2e34c\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " Oct 30 14:13:50.634393 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.634214 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-token\" (UniqueName: \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-token\") pod \"366ae49d-567d-46c9-a21f-07a796f2e34c\" (UID: \"366ae49d-567d-46c9-a21f-07a796f2e34c\") " Oct 30 14:13:50.634819 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.634776 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-ca-configmap" pod "366ae49d-567d-46c9-a21f-07a796f2e34c" (UID: "366ae49d-567d-46c9-a21f-07a796f2e34c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:50.637063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.637030 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-kube-api-access-mn6qv" pod "366ae49d-567d-46c9-a21f-07a796f2e34c" (UID: "366ae49d-567d-46c9-a21f-07a796f2e34c"). InnerVolumeSpecName "kube-api-access-mn6qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:50.637330 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.637300 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-dns-cert" pod "366ae49d-567d-46c9-a21f-07a796f2e34c" (UID: "366ae49d-567d-46c9-a21f-07a796f2e34c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:50.637432 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.637323 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-cacerts" pod "366ae49d-567d-46c9-a21f-07a796f2e34c" (UID: "366ae49d-567d-46c9-a21f-07a796f2e34c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:50.637432 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.637384 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-token" pod "366ae49d-567d-46c9-a21f-07a796f2e34c" (UID: "366ae49d-567d-46c9-a21f-07a796f2e34c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:50.637526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.637442 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366ae49d-567d-46c9-a21f-07a796f2e34c-local-certs" pod "366ae49d-567d-46c9-a21f-07a796f2e34c" (UID: "366ae49d-567d-46c9-a21f-07a796f2e34c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:50.637526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.637445 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-kubeconfig" pod "366ae49d-567d-46c9-a21f-07a796f2e34c" (UID: "366ae49d-567d-46c9-a21f-07a796f2e34c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:50.734957 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.734926 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn6qv\" (UniqueName: \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-kube-api-access-mn6qv\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:13:50.734957 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.734953 4968 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-ca-configmap\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:13:50.734957 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.734964 4968 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-cacerts\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:13:50.735168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.734975 4968 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-csr-dns-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:13:50.735168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.734984 4968 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-kubeconfig\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:13:50.735168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.734993 4968 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/366ae49d-567d-46c9-a21f-07a796f2e34c-local-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:13:50.735168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:50.735002 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366ae49d-567d-46c9-a21f-07a796f2e34c-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:13:51.285795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:51.285761 4968 generic.go:356] "Generic (PLEG): container finished" podID="366ae49d-567d-46c9-a21f-07a796f2e34c" containerID="24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289" exitCode=0 Oct 30 14:13:51.286257 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:51.285814 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" Oct 30 14:13:51.286257 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:51.285850 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" event={"ID":"366ae49d-567d-46c9-a21f-07a796f2e34c","Type":"ContainerDied","Data":"24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289"} Oct 30 14:13:51.286257 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:51.285910 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2" event={"ID":"366ae49d-567d-46c9-a21f-07a796f2e34c","Type":"ContainerDied","Data":"f7cdfa1f2bfdeed2fcb712ff750b556e214e078355df07f158e21e7aaa7faccf"} Oct 30 14:13:51.286257 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:51.285940 4968 scope.go:122] "RemoveContainer" containerID="24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289" Oct 30 14:13:51.296489 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:51.296402 4968 scope.go:122] "RemoveContainer" containerID="24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289" Oct 30 14:13:51.296827 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:13:51.296799 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289\": container with ID starting with 24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289 not found: ID does not exist" containerID="24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289" Oct 30 14:13:51.296925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:51.296840 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289"} err="failed to get container status \"24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289\": rpc error: code = NotFound desc = could not find container \"24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289\": container with ID starting with 24f894370139798308a4863aebdce5720db054808dee3cbfa74248e6a168a289 not found: ID does not exist" Oct 30 14:13:51.342032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:51.341999 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2"] Oct 30 14:13:51.351912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:51.351892 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96zq2"] Oct 30 14:13:52.517062 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:13:52.517029 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366ae49d-567d-46c9-a21f-07a796f2e34c" path="/var/lib/kubelet/pods/366ae49d-567d-46c9-a21f-07a796f2e34c/volumes" Oct 30 14:14:20.443371 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:14:20.443335 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:14:50.446287 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:14:50.446248 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:15:20.443971 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:15:20.443928 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:15:24.336615 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.336578 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-cmbdv"] Oct 30 14:15:24.337139 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.337017 4968 cpu_manager.go:414] "containerMap: removing container" podUID="366ae49d-567d-46c9-a21f-07a796f2e34c" containerName="discovery" Oct 30 14:15:24.337139 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.337033 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366ae49d-567d-46c9-a21f-07a796f2e34c" containerName="discovery" Oct 30 14:15:24.337139 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.337093 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="366ae49d-567d-46c9-a21f-07a796f2e34c" containerName="discovery" Oct 30 14:15:24.340173 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.340150 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-cmbdv" Oct 30 14:15:24.358185 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.358157 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 14:15:24.358294 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.358215 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 14:15:24.363702 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.363677 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-z6nmb\"" Oct 30 14:15:24.365272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.365246 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-cmbdv"] Oct 30 14:15:24.475961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.475933 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxpf\" (UniqueName: \"kubernetes.io/projected/64121261-495b-4cfc-855a-edf6f69d62d0-kube-api-access-9pxpf\") pod \"authorino-operator-68d6dd76dc-cmbdv\" (UID: \"64121261-495b-4cfc-855a-edf6f69d62d0\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-cmbdv" Oct 30 14:15:24.576816 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.576782 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxpf\" (UniqueName: \"kubernetes.io/projected/64121261-495b-4cfc-855a-edf6f69d62d0-kube-api-access-9pxpf\") pod \"authorino-operator-68d6dd76dc-cmbdv\" (UID: \"64121261-495b-4cfc-855a-edf6f69d62d0\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-cmbdv" Oct 30 14:15:24.600099 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.600043 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxpf\" (UniqueName: \"kubernetes.io/projected/64121261-495b-4cfc-855a-edf6f69d62d0-kube-api-access-9pxpf\") pod \"authorino-operator-68d6dd76dc-cmbdv\" (UID: \"64121261-495b-4cfc-855a-edf6f69d62d0\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-cmbdv" Oct 30 14:15:24.652567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.652540 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-cmbdv" Oct 30 14:15:24.954927 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:24.954850 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-cmbdv"] Oct 30 14:15:24.958520 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:15:24.958496 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64121261_495b_4cfc_855a_edf6f69d62d0.slice/crio-fc9534a5b915b3acca9634d7c4ad284bcc8f8b03d0a3177f9c6b76ef523b40ae WatchSource:0}: Error finding container fc9534a5b915b3acca9634d7c4ad284bcc8f8b03d0a3177f9c6b76ef523b40ae: Status 404 returned error can't find the container with id fc9534a5b915b3acca9634d7c4ad284bcc8f8b03d0a3177f9c6b76ef523b40ae Oct 30 14:15:25.668810 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:25.668730 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-cmbdv" event={"ID":"64121261-495b-4cfc-855a-edf6f69d62d0","Type":"ContainerStarted","Data":"fc9534a5b915b3acca9634d7c4ad284bcc8f8b03d0a3177f9c6b76ef523b40ae"} Oct 30 14:15:26.604419 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:26.603399 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6"] Oct 30 14:15:26.613633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:26.613600 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6" Oct 30 14:15:26.630380 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:26.630356 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zm5rw\"" Oct 30 14:15:26.640538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:26.640506 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 14:15:26.642391 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:26.642366 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6"] Oct 30 14:15:26.696654 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:26.696625 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnf8\" (UniqueName: \"kubernetes.io/projected/ad114249-10f3-4069-b4ee-86543a6e1c46-kube-api-access-ttnf8\") pod \"dns-operator-controller-manager-65bfd79598-x8hl6\" (UID: \"ad114249-10f3-4069-b4ee-86543a6e1c46\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6" Oct 30 14:15:26.797376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:26.797331 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnf8\" (UniqueName: \"kubernetes.io/projected/ad114249-10f3-4069-b4ee-86543a6e1c46-kube-api-access-ttnf8\") pod \"dns-operator-controller-manager-65bfd79598-x8hl6\" (UID: \"ad114249-10f3-4069-b4ee-86543a6e1c46\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6" Oct 30 14:15:26.820998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:26.820960 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnf8\" (UniqueName: \"kubernetes.io/projected/ad114249-10f3-4069-b4ee-86543a6e1c46-kube-api-access-ttnf8\") pod \"dns-operator-controller-manager-65bfd79598-x8hl6\" (UID: \"ad114249-10f3-4069-b4ee-86543a6e1c46\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6" Oct 30 14:15:26.928902 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:26.928835 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6" Oct 30 14:15:27.277751 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:27.277630 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6"] Oct 30 14:15:27.282292 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:15:27.282262 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad114249_10f3_4069_b4ee_86543a6e1c46.slice/crio-3e2095defc88b4b4305eb143bec635d5e9c0cf316b70721da50c83cbe68f46b3 WatchSource:0}: Error finding container 3e2095defc88b4b4305eb143bec635d5e9c0cf316b70721da50c83cbe68f46b3: Status 404 returned error can't find the container with id 3e2095defc88b4b4305eb143bec635d5e9c0cf316b70721da50c83cbe68f46b3 Oct 30 14:15:27.679071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:27.679037 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6" event={"ID":"ad114249-10f3-4069-b4ee-86543a6e1c46","Type":"ContainerStarted","Data":"3e2095defc88b4b4305eb143bec635d5e9c0cf316b70721da50c83cbe68f46b3"} Oct 30 14:15:27.680665 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:27.680632 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-cmbdv" event={"ID":"64121261-495b-4cfc-855a-edf6f69d62d0","Type":"ContainerStarted","Data":"3c51d0535b054a36391b5a69ee713e006da4f527525ff969b3b667127d3af7c1"} Oct 30 14:15:27.680838 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:27.680751 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-cmbdv" Oct 30 14:15:27.720252 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:27.720202 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-cmbdv" podStartSLOduration=1.393353458 podStartE2EDuration="3.720189382s" totalImagesPullingTime="2.326835924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:27.718921203 +0000 UTC m=+523.344364568" watchObservedRunningTime="2026-10-30 14:15:27.720189382 +0000 UTC m=+523.345632754" Oct 30 14:15:28.687767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:28.687739 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6" event={"ID":"ad114249-10f3-4069-b4ee-86543a6e1c46","Type":"ContainerStarted","Data":"9f43a173c34213dd05e49fda61867973c0cd00196a4bb6bca8a5c6e775ffb47e"} Oct 30 14:15:28.687888 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:28.687796 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6" Oct 30 14:15:28.741599 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:28.741541 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6" podStartSLOduration=1.398238915 podStartE2EDuration="2.741525563s" totalImagesPullingTime="1.343286648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:28.740510266 +0000 UTC m=+524.365953640" watchObservedRunningTime="2026-10-30 14:15:28.741525563 +0000 UTC m=+524.366968936" Oct 30 14:15:33.248620 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.248584 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8"] Oct 30 14:15:33.252046 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.252029 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:33.267137 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.267118 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wnlf4\"" Oct 30 14:15:33.278767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.278743 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8"] Oct 30 14:15:33.351886 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.351857 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-d6zs8\" (UID: \"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:33.352018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.351918 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r92zf\" (UniqueName: \"kubernetes.io/projected/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-kube-api-access-r92zf\") pod \"kuadrant-operator-controller-manager-6b6fb56846-d6zs8\" (UID: \"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:33.452843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.452799 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r92zf\" (UniqueName: \"kubernetes.io/projected/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-kube-api-access-r92zf\") pod \"kuadrant-operator-controller-manager-6b6fb56846-d6zs8\" (UID: \"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:33.452995 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.452893 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-d6zs8\" (UID: \"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:33.453250 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.453231 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-d6zs8\" (UID: \"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:33.476804 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.476774 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r92zf\" (UniqueName: \"kubernetes.io/projected/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-kube-api-access-r92zf\") pod \"kuadrant-operator-controller-manager-6b6fb56846-d6zs8\" (UID: \"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:33.563898 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.563821 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:33.896621 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:33.896584 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8"] Oct 30 14:15:33.901133 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:15:33.901109 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27d7d6f_6f42_4d95_b236_8bea5fcf2ac7.slice/crio-0f6a8bc50783750346cd2d262540162e04ee8c7d40d9431f37d663779f5afb8b WatchSource:0}: Error finding container 0f6a8bc50783750346cd2d262540162e04ee8c7d40d9431f37d663779f5afb8b: Status 404 returned error can't find the container with id 0f6a8bc50783750346cd2d262540162e04ee8c7d40d9431f37d663779f5afb8b Oct 30 14:15:34.715162 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:34.715128 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" event={"ID":"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7","Type":"ContainerStarted","Data":"0f6a8bc50783750346cd2d262540162e04ee8c7d40d9431f37d663779f5afb8b"} Oct 30 14:15:37.730477 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:37.730443 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" event={"ID":"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7","Type":"ContainerStarted","Data":"d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb"} Oct 30 14:15:37.730902 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:37.730547 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:37.793761 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:37.793655 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" podStartSLOduration=1.204644026 podStartE2EDuration="4.793642298s" totalImagesPullingTime="3.588998272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:37.782180388 +0000 UTC m=+533.407623760" watchObservedRunningTime="2026-10-30 14:15:37.793642298 +0000 UTC m=+533.419085673" Oct 30 14:15:38.690281 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:38.690250 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-cmbdv" Oct 30 14:15:39.694612 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:39.694581 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x8hl6" Oct 30 14:15:48.737197 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:48.737168 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:50.026514 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.026473 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx"] Oct 30 14:15:50.030219 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.030197 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:50.054356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.054332 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx"] Oct 30 14:15:50.194365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.194337 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2f4q\" (UniqueName: \"kubernetes.io/projected/55d044ea-218e-4980-8ec6-a4b92111930d-kube-api-access-w2f4q\") pod \"kuadrant-operator-controller-manager-6b76784d48-vznqx\" (UID: \"55d044ea-218e-4980-8ec6-a4b92111930d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:50.194509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.194383 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55d044ea-218e-4980-8ec6-a4b92111930d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-vznqx\" (UID: \"55d044ea-218e-4980-8ec6-a4b92111930d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:50.295240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.295168 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f4q\" (UniqueName: \"kubernetes.io/projected/55d044ea-218e-4980-8ec6-a4b92111930d-kube-api-access-w2f4q\") pod \"kuadrant-operator-controller-manager-6b76784d48-vznqx\" (UID: \"55d044ea-218e-4980-8ec6-a4b92111930d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:50.295240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.295215 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55d044ea-218e-4980-8ec6-a4b92111930d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-vznqx\" (UID: \"55d044ea-218e-4980-8ec6-a4b92111930d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:50.295540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.295525 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55d044ea-218e-4980-8ec6-a4b92111930d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-vznqx\" (UID: \"55d044ea-218e-4980-8ec6-a4b92111930d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:50.317826 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.317800 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2f4q\" (UniqueName: \"kubernetes.io/projected/55d044ea-218e-4980-8ec6-a4b92111930d-kube-api-access-w2f4q\") pod \"kuadrant-operator-controller-manager-6b76784d48-vznqx\" (UID: \"55d044ea-218e-4980-8ec6-a4b92111930d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:50.343150 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.343117 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:50.443078 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:15:50.443042 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:15:50.667363 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.667323 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx"] Oct 30 14:15:50.671814 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:15:50.671786 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d044ea_218e_4980_8ec6_a4b92111930d.slice/crio-a3020a98eb7bb376178d78e9a373ee30ca952814317f5e6082d8f959fc6e7bd8 WatchSource:0}: Error finding container a3020a98eb7bb376178d78e9a373ee30ca952814317f5e6082d8f959fc6e7bd8: Status 404 returned error can't find the container with id a3020a98eb7bb376178d78e9a373ee30ca952814317f5e6082d8f959fc6e7bd8 Oct 30 14:15:50.788553 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.788517 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" event={"ID":"55d044ea-218e-4980-8ec6-a4b92111930d","Type":"ContainerStarted","Data":"bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746"} Oct 30 14:15:50.788736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.788573 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" event={"ID":"55d044ea-218e-4980-8ec6-a4b92111930d","Type":"ContainerStarted","Data":"a3020a98eb7bb376178d78e9a373ee30ca952814317f5e6082d8f959fc6e7bd8"} Oct 30 14:15:50.788736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.788635 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:50.832925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:50.832875 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" podStartSLOduration=1.832862685 podStartE2EDuration="1.832862685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:50.832652653 +0000 UTC m=+546.458096027" watchObservedRunningTime="2026-10-30 14:15:50.832862685 +0000 UTC m=+546.458306055" Oct 30 14:15:52.263503 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.263466 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8"] Oct 30 14:15:52.263988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.263652 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" podUID="f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" containerName="manager" containerID="cri-o://d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb" gracePeriod=2 Oct 30 14:15:52.264784 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.264752 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8"] Oct 30 14:15:52.275595 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:15:52.275565 4968 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-d6zs8\" is forbidden: User \"system:node:ip-10-0-129-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-158.ec2.internal' and this object" podUID="f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:52.284918 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.284892 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs"] Oct 30 14:15:52.285378 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.285357 4968 cpu_manager.go:414] "containerMap: removing container" podUID="f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" containerName="manager" Oct 30 14:15:52.285470 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.285381 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" containerName="manager" Oct 30 14:15:52.285530 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.285504 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" containerName="manager" Oct 30 14:15:52.288498 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.288480 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx"] Oct 30 14:15:52.288606 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.288589 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:15:52.293892 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.293868 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx"] Oct 30 14:15:52.299954 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:15:52.299920 4968 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-d6zs8\" is forbidden: User \"system:node:ip-10-0-129-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-158.ec2.internal' and this object" podUID="f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:52.315178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.315143 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs"] Oct 30 14:15:52.318213 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.318188 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc"] Oct 30 14:15:52.320196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.318983 4968 cpu_manager.go:414] "containerMap: removing container" podUID="55d044ea-218e-4980-8ec6-a4b92111930d" containerName="manager" Oct 30 14:15:52.320196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.319005 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d044ea-218e-4980-8ec6-a4b92111930d" containerName="manager" Oct 30 14:15:52.320196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.319119 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d044ea-218e-4980-8ec6-a4b92111930d" containerName="manager" Oct 30 14:15:52.322293 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.322275 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:15:52.358504 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.358138 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc"] Oct 30 14:15:52.360326 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:15:52.360300 4968 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-d6zs8\" is forbidden: User \"system:node:ip-10-0-129-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-158.ec2.internal' and this object" podUID="f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:52.413564 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.413529 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhfd\" (UniqueName: \"kubernetes.io/projected/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-kube-api-access-vnhfd\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ckgfs\" (UID: \"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:15:52.413731 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.413599 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ckgfs\" (UID: \"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:15:52.513555 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.513503 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:52.514240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.514218 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ckgfs\" (UID: \"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:15:52.514310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.514284 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b97f36-44bb-4f40-96b7-98ab13848c15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-dz7zc\" (UID: \"67b97f36-44bb-4f40-96b7-98ab13848c15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:15:52.514353 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.514324 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhfd\" (UniqueName: \"kubernetes.io/projected/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-kube-api-access-vnhfd\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ckgfs\" (UID: \"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:15:52.514412 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.514391 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btr9j\" (UniqueName: \"kubernetes.io/projected/67b97f36-44bb-4f40-96b7-98ab13848c15-kube-api-access-btr9j\") pod \"kuadrant-operator-controller-manager-6b76784d48-dz7zc\" (UID: \"67b97f36-44bb-4f40-96b7-98ab13848c15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:15:52.514551 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.514534 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ckgfs\" (UID: \"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:15:52.525037 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:15:52.525009 4968 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-d6zs8\" is forbidden: User \"system:node:ip-10-0-129-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-158.ec2.internal' and this object" podUID="f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:52.546273 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.546245 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhfd\" (UniqueName: \"kubernetes.io/projected/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-kube-api-access-vnhfd\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ckgfs\" (UID: \"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:15:52.614889 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.614865 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-extensions-socket-volume\") pod \"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7\" (UID: \"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7\") " Oct 30 14:15:52.615016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.614954 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-kube-api-access-r92zf\" (UniqueName: \"kubernetes.io/projected/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-kube-api-access-r92zf\") pod \"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7\" (UID: \"f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7\") " Oct 30 14:15:52.615117 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.615098 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b97f36-44bb-4f40-96b7-98ab13848c15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-dz7zc\" (UID: \"67b97f36-44bb-4f40-96b7-98ab13848c15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:15:52.615181 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.615165 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btr9j\" (UniqueName: \"kubernetes.io/projected/67b97f36-44bb-4f40-96b7-98ab13848c15-kube-api-access-btr9j\") pod \"kuadrant-operator-controller-manager-6b76784d48-dz7zc\" (UID: \"67b97f36-44bb-4f40-96b7-98ab13848c15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:15:52.615309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.615290 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-extensions-socket-volume" pod "f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" (UID: "f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:15:52.615447 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.615427 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b97f36-44bb-4f40-96b7-98ab13848c15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-dz7zc\" (UID: \"67b97f36-44bb-4f40-96b7-98ab13848c15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:15:52.617126 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.617107 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-kube-api-access-r92zf" pod "f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" (UID: "f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7"). InnerVolumeSpecName "kube-api-access-r92zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:15:52.638336 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.638311 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btr9j\" (UniqueName: \"kubernetes.io/projected/67b97f36-44bb-4f40-96b7-98ab13848c15-kube-api-access-btr9j\") pod \"kuadrant-operator-controller-manager-6b76784d48-dz7zc\" (UID: \"67b97f36-44bb-4f40-96b7-98ab13848c15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:15:52.658161 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.658140 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:15:52.668394 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.668370 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:15:52.720375 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.716724 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r92zf\" (UniqueName: \"kubernetes.io/projected/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-kube-api-access-r92zf\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:15:52.720375 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.716753 4968 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7-extensions-socket-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:15:52.798462 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.798383 4968 generic.go:356] "Generic (PLEG): container finished" podID="f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" containerID="d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb" exitCode=0 Oct 30 14:15:52.798462 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.798434 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-d6zs8" Oct 30 14:15:52.798641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.798481 4968 scope.go:122] "RemoveContainer" containerID="d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb" Oct 30 14:15:52.798746 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.798695 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" podUID="55d044ea-218e-4980-8ec6-a4b92111930d" containerName="manager" containerID="cri-o://bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746" gracePeriod=2 Oct 30 14:15:52.808341 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.808321 4968 scope.go:122] "RemoveContainer" containerID="d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb" Oct 30 14:15:52.808606 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:15:52.808584 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb\": container with ID starting with d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb not found: ID does not exist" containerID="d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb" Oct 30 14:15:52.808667 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.808615 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb"} err="failed to get container status \"d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb\": rpc error: code = NotFound desc = could not find container \"d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb\": container with ID starting with d26cc99368604a2c371c8fb82c5121b5a1e657b049b60c18fa262742652f76eb not found: ID does not exist" Oct 30 14:15:52.980271 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.980211 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs"] Oct 30 14:15:52.984129 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:15:52.984090 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74dbd4cd_1bd0_4c62_bf33_dbf773b8dbad.slice/crio-cd61fa401fcda65060290b79682dd506c241a6bd86e8a3909d4cca1a0ee58c06 WatchSource:0}: Error finding container cd61fa401fcda65060290b79682dd506c241a6bd86e8a3909d4cca1a0ee58c06: Status 404 returned error can't find the container with id cd61fa401fcda65060290b79682dd506c241a6bd86e8a3909d4cca1a0ee58c06 Oct 30 14:15:52.992986 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:52.992917 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc"] Oct 30 14:15:52.996997 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:15:52.996970 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b97f36_44bb_4f40_96b7_98ab13848c15.slice/crio-7ef8e2ff91d51e25c110699adff2163de26ede76098aabb403fef3c21ce84c68 WatchSource:0}: Error finding container 7ef8e2ff91d51e25c110699adff2163de26ede76098aabb403fef3c21ce84c68: Status 404 returned error can't find the container with id 7ef8e2ff91d51e25c110699adff2163de26ede76098aabb403fef3c21ce84c68 Oct 30 14:15:53.061286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.061263 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:53.220963 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.220933 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d044ea-218e-4980-8ec6-a4b92111930d-kube-api-access-w2f4q\" (UniqueName: \"kubernetes.io/projected/55d044ea-218e-4980-8ec6-a4b92111930d-kube-api-access-w2f4q\") pod \"55d044ea-218e-4980-8ec6-a4b92111930d\" (UID: \"55d044ea-218e-4980-8ec6-a4b92111930d\") " Oct 30 14:15:53.221133 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.221006 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d044ea-218e-4980-8ec6-a4b92111930d-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55d044ea-218e-4980-8ec6-a4b92111930d-extensions-socket-volume\") pod \"55d044ea-218e-4980-8ec6-a4b92111930d\" (UID: \"55d044ea-218e-4980-8ec6-a4b92111930d\") " Oct 30 14:15:53.221367 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.221341 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d044ea-218e-4980-8ec6-a4b92111930d-extensions-socket-volume" pod "55d044ea-218e-4980-8ec6-a4b92111930d" (UID: "55d044ea-218e-4980-8ec6-a4b92111930d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:15:53.223027 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.223003 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d044ea-218e-4980-8ec6-a4b92111930d-kube-api-access-w2f4q" pod "55d044ea-218e-4980-8ec6-a4b92111930d" (UID: "55d044ea-218e-4980-8ec6-a4b92111930d"). InnerVolumeSpecName "kube-api-access-w2f4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:15:53.322192 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.322110 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2f4q\" (UniqueName: \"kubernetes.io/projected/55d044ea-218e-4980-8ec6-a4b92111930d-kube-api-access-w2f4q\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:15:53.322192 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.322152 4968 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55d044ea-218e-4980-8ec6-a4b92111930d-extensions-socket-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:15:53.804359 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.804313 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" event={"ID":"67b97f36-44bb-4f40-96b7-98ab13848c15","Type":"ContainerStarted","Data":"292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c"} Oct 30 14:15:53.804547 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.804382 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:15:53.804547 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.804405 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" event={"ID":"67b97f36-44bb-4f40-96b7-98ab13848c15","Type":"ContainerStarted","Data":"7ef8e2ff91d51e25c110699adff2163de26ede76098aabb403fef3c21ce84c68"} Oct 30 14:15:53.806008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.805983 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" event={"ID":"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad","Type":"ContainerStarted","Data":"0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07"} Oct 30 14:15:53.806120 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.806023 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" event={"ID":"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad","Type":"ContainerStarted","Data":"cd61fa401fcda65060290b79682dd506c241a6bd86e8a3909d4cca1a0ee58c06"} Oct 30 14:15:53.806120 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.806057 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:15:53.807312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.807289 4968 generic.go:356] "Generic (PLEG): container finished" podID="55d044ea-218e-4980-8ec6-a4b92111930d" containerID="bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746" exitCode=0 Oct 30 14:15:53.807371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.807317 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:53.807410 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.807381 4968 scope.go:122] "RemoveContainer" containerID="bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746" Oct 30 14:15:53.816287 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.816271 4968 scope.go:122] "RemoveContainer" containerID="bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746" Oct 30 14:15:53.816578 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:15:53.816559 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746\": container with ID starting with bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746 not found: ID does not exist" containerID="bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746" Oct 30 14:15:53.816634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.816587 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746"} err="failed to get container status \"bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746\": rpc error: code = NotFound desc = could not find container \"bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746\": container with ID starting with bf6cd083fa91e365ac25ab4aed067d450b35a899d19ad1a47e36a4c0c213c746 not found: ID does not exist" Oct 30 14:15:53.892001 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.891954 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" podStartSLOduration=1.891940937 podStartE2EDuration="1.891940937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:53.881857534 +0000 UTC m=+549.507300918" watchObservedRunningTime="2026-10-30 14:15:53.891940937 +0000 UTC m=+549.517384310" Oct 30 14:15:53.899038 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:15:53.899011 4968 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-vznqx\" is forbidden: User \"system:node:ip-10-0-129-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-158.ec2.internal' and this object" podUID="55d044ea-218e-4980-8ec6-a4b92111930d" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:15:53.955033 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:53.954985 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" podStartSLOduration=1.954971848 podStartE2EDuration="1.954971848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:53.953580096 +0000 UTC m=+549.579023468" watchObservedRunningTime="2026-10-30 14:15:53.954971848 +0000 UTC m=+549.580415221" Oct 30 14:15:54.517001 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:54.516970 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d044ea-218e-4980-8ec6-a4b92111930d" path="/var/lib/kubelet/pods/55d044ea-218e-4980-8ec6-a4b92111930d/volumes" Oct 30 14:15:54.517415 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:15:54.517400 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7" path="/var/lib/kubelet/pods/f27d7d6f-6f42-4d95-b236-8bea5fcf2ac7/volumes" Oct 30 14:15:54.524943 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:15:54.524921 4968 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-vznqx\" is forbidden: User \"system:node:ip-10-0-129-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-158.ec2.internal' and this object" podUID="55d044ea-218e-4980-8ec6-a4b92111930d" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vznqx" Oct 30 14:16:04.816362 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:04.816331 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:16:04.816872 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:04.816614 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:16:04.955122 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:04.955082 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs"] Oct 30 14:16:04.955333 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:04.955305 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" podUID="74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad" containerName="manager" containerID="cri-o://0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07" gracePeriod=10 Oct 30 14:16:05.192509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.192482 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:16:05.214107 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.214083 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-extensions-socket-volume\") pod \"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad\" (UID: \"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad\") " Oct 30 14:16:05.214210 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.214146 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-kube-api-access-vnhfd\" (UniqueName: \"kubernetes.io/projected/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-kube-api-access-vnhfd\") pod \"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad\" (UID: \"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad\") " Oct 30 14:16:05.214457 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.214431 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-extensions-socket-volume" pod "74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad" (UID: "74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:16:05.216182 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.216152 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-kube-api-access-vnhfd" pod "74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad" (UID: "74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad"). InnerVolumeSpecName "kube-api-access-vnhfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:16:05.315577 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.315548 4968 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-extensions-socket-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:16:05.315577 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.315571 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnhfd\" (UniqueName: \"kubernetes.io/projected/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad-kube-api-access-vnhfd\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:16:05.359012 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.358819 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs"] Oct 30 14:16:05.359480 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.359462 4968 cpu_manager.go:414] "containerMap: removing container" podUID="74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad" containerName="manager" Oct 30 14:16:05.359554 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.359481 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad" containerName="manager" Oct 30 14:16:05.359599 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.359564 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad" containerName="manager" Oct 30 14:16:05.362462 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.362444 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:16:05.397239 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.397213 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs"] Oct 30 14:16:05.416927 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.416902 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be2a30d6-be3f-4247-8223-714a89309f07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-q79hs\" (UID: \"be2a30d6-be3f-4247-8223-714a89309f07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:16:05.417024 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.416964 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvxk5\" (UniqueName: \"kubernetes.io/projected/be2a30d6-be3f-4247-8223-714a89309f07-kube-api-access-nvxk5\") pod \"kuadrant-operator-controller-manager-57bb9bfff-q79hs\" (UID: \"be2a30d6-be3f-4247-8223-714a89309f07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:16:05.517674 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.517634 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be2a30d6-be3f-4247-8223-714a89309f07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-q79hs\" (UID: \"be2a30d6-be3f-4247-8223-714a89309f07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:16:05.517866 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.517688 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvxk5\" (UniqueName: \"kubernetes.io/projected/be2a30d6-be3f-4247-8223-714a89309f07-kube-api-access-nvxk5\") pod \"kuadrant-operator-controller-manager-57bb9bfff-q79hs\" (UID: \"be2a30d6-be3f-4247-8223-714a89309f07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:16:05.518074 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.518053 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be2a30d6-be3f-4247-8223-714a89309f07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-q79hs\" (UID: \"be2a30d6-be3f-4247-8223-714a89309f07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:16:05.546370 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.546336 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvxk5\" (UniqueName: \"kubernetes.io/projected/be2a30d6-be3f-4247-8223-714a89309f07-kube-api-access-nvxk5\") pod \"kuadrant-operator-controller-manager-57bb9bfff-q79hs\" (UID: \"be2a30d6-be3f-4247-8223-714a89309f07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:16:05.674584 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.674501 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:16:05.867511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.867472 4968 generic.go:356] "Generic (PLEG): container finished" podID="74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad" containerID="0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07" exitCode=0 Oct 30 14:16:05.867511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.867512 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" Oct 30 14:16:05.868025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.867504 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" event={"ID":"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad","Type":"ContainerDied","Data":"0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07"} Oct 30 14:16:05.868025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.867632 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs" event={"ID":"74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad","Type":"ContainerDied","Data":"cd61fa401fcda65060290b79682dd506c241a6bd86e8a3909d4cca1a0ee58c06"} Oct 30 14:16:05.868025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.867658 4968 scope.go:122] "RemoveContainer" containerID="0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07" Oct 30 14:16:05.877552 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.877537 4968 scope.go:122] "RemoveContainer" containerID="0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07" Oct 30 14:16:05.877824 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:16:05.877807 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07\": container with ID starting with 0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07 not found: ID does not exist" containerID="0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07" Oct 30 14:16:05.877877 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.877833 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07"} err="failed to get container status \"0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07\": rpc error: code = NotFound desc = could not find container \"0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07\": container with ID starting with 0b6f464eace3b4b09d7935db288822539d06ddcf5ab71a2215df55fe63a3ef07 not found: ID does not exist" Oct 30 14:16:05.919279 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.919249 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs"] Oct 30 14:16:05.929684 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.929631 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ckgfs"] Oct 30 14:16:05.989309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:05.989279 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs"] Oct 30 14:16:05.993587 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:16:05.993562 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2a30d6_be3f_4247_8223_714a89309f07.slice/crio-7c3e088423f1abca82498a28f7df1f3f0df1b3e1bf2943ccaf82aef2a9a2b66b WatchSource:0}: Error finding container 7c3e088423f1abca82498a28f7df1f3f0df1b3e1bf2943ccaf82aef2a9a2b66b: Status 404 returned error can't find the container with id 7c3e088423f1abca82498a28f7df1f3f0df1b3e1bf2943ccaf82aef2a9a2b66b Oct 30 14:16:06.517242 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:06.517203 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad" path="/var/lib/kubelet/pods/74dbd4cd-1bd0-4c62-bf33-dbf773b8dbad/volumes" Oct 30 14:16:06.874354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:06.874317 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" event={"ID":"be2a30d6-be3f-4247-8223-714a89309f07","Type":"ContainerStarted","Data":"74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa"} Oct 30 14:16:06.874814 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:06.874385 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" event={"ID":"be2a30d6-be3f-4247-8223-714a89309f07","Type":"ContainerStarted","Data":"7c3e088423f1abca82498a28f7df1f3f0df1b3e1bf2943ccaf82aef2a9a2b66b"} Oct 30 14:16:06.874814 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:06.874450 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:16:06.925226 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:06.925186 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" podStartSLOduration=1.925175586 podStartE2EDuration="1.925175586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:06.924931565 +0000 UTC m=+562.550374939" watchObservedRunningTime="2026-10-30 14:16:06.925175586 +0000 UTC m=+562.550618958" Oct 30 14:16:17.882823 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:17.882788 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:16:18.011257 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.011222 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc"] Oct 30 14:16:18.011512 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.011485 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" podUID="67b97f36-44bb-4f40-96b7-98ab13848c15" containerName="manager" containerID="cri-o://292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c" gracePeriod=10 Oct 30 14:16:18.258826 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.258803 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:16:18.324679 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.324649 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b97f36-44bb-4f40-96b7-98ab13848c15-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b97f36-44bb-4f40-96b7-98ab13848c15-extensions-socket-volume\") pod \"67b97f36-44bb-4f40-96b7-98ab13848c15\" (UID: \"67b97f36-44bb-4f40-96b7-98ab13848c15\") " Oct 30 14:16:18.324679 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.324681 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b97f36-44bb-4f40-96b7-98ab13848c15-kube-api-access-btr9j\" (UniqueName: \"kubernetes.io/projected/67b97f36-44bb-4f40-96b7-98ab13848c15-kube-api-access-btr9j\") pod \"67b97f36-44bb-4f40-96b7-98ab13848c15\" (UID: \"67b97f36-44bb-4f40-96b7-98ab13848c15\") " Oct 30 14:16:18.325019 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.324995 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b97f36-44bb-4f40-96b7-98ab13848c15-extensions-socket-volume" pod "67b97f36-44bb-4f40-96b7-98ab13848c15" (UID: "67b97f36-44bb-4f40-96b7-98ab13848c15"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:16:18.326711 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.326684 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b97f36-44bb-4f40-96b7-98ab13848c15-kube-api-access-btr9j" pod "67b97f36-44bb-4f40-96b7-98ab13848c15" (UID: "67b97f36-44bb-4f40-96b7-98ab13848c15"). InnerVolumeSpecName "kube-api-access-btr9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:16:18.425653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.425621 4968 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b97f36-44bb-4f40-96b7-98ab13848c15-extensions-socket-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:16:18.425653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.425650 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btr9j\" (UniqueName: \"kubernetes.io/projected/67b97f36-44bb-4f40-96b7-98ab13848c15-kube-api-access-btr9j\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:16:18.929908 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.929873 4968 generic.go:356] "Generic (PLEG): container finished" podID="67b97f36-44bb-4f40-96b7-98ab13848c15" containerID="292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c" exitCode=0 Oct 30 14:16:18.930320 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.929920 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" event={"ID":"67b97f36-44bb-4f40-96b7-98ab13848c15","Type":"ContainerDied","Data":"292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c"} Oct 30 14:16:18.930320 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.929940 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" Oct 30 14:16:18.930320 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.929969 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc" event={"ID":"67b97f36-44bb-4f40-96b7-98ab13848c15","Type":"ContainerDied","Data":"7ef8e2ff91d51e25c110699adff2163de26ede76098aabb403fef3c21ce84c68"} Oct 30 14:16:18.930320 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.930000 4968 scope.go:122] "RemoveContainer" containerID="292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c" Oct 30 14:16:18.938238 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.938216 4968 scope.go:122] "RemoveContainer" containerID="292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c" Oct 30 14:16:18.938518 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:16:18.938496 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c\": container with ID starting with 292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c not found: ID does not exist" containerID="292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c" Oct 30 14:16:18.938564 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.938527 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c"} err="failed to get container status \"292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c\": rpc error: code = NotFound desc = could not find container \"292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c\": container with ID starting with 292afa863efdd173acbbd619a16061f7e94764a66bf47666fb69b01587d95c2c not found: ID does not exist" Oct 30 14:16:18.984088 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.984061 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc"] Oct 30 14:16:18.994003 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:18.993983 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dz7zc"] Oct 30 14:16:20.443555 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:16:20.443454 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:16:20.517318 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:20.517276 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b97f36-44bb-4f40-96b7-98ab13848c15" path="/var/lib/kubelet/pods/67b97f36-44bb-4f40-96b7-98ab13848c15/volumes" Oct 30 14:16:37.029515 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.029300 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:16:37.030445 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.030419 4968 cpu_manager.go:414] "containerMap: removing container" podUID="67b97f36-44bb-4f40-96b7-98ab13848c15" containerName="manager" Oct 30 14:16:37.030445 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.030442 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b97f36-44bb-4f40-96b7-98ab13848c15" containerName="manager" Oct 30 14:16:37.030590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.030510 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b97f36-44bb-4f40-96b7-98ab13848c15" containerName="manager" Oct 30 14:16:37.033668 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.033650 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" Oct 30 14:16:37.047135 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.047112 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 14:16:37.057199 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.057169 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bn8rh\"" Oct 30 14:16:37.058924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.058900 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:16:37.127229 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.127183 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:16:37.183158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.183128 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e8e6e17d-44a6-4e4e-bba4-236c02d5a856-config-file\") pod \"limitador-limitador-7c9b87d574-svfmk\" (UID: \"e8e6e17d-44a6-4e4e-bba4-236c02d5a856\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" Oct 30 14:16:37.183314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.183172 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2nnw\" (UniqueName: \"kubernetes.io/projected/e8e6e17d-44a6-4e4e-bba4-236c02d5a856-kube-api-access-j2nnw\") pod \"limitador-limitador-7c9b87d574-svfmk\" (UID: \"e8e6e17d-44a6-4e4e-bba4-236c02d5a856\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" Oct 30 14:16:37.284518 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.284441 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e8e6e17d-44a6-4e4e-bba4-236c02d5a856-config-file\") pod \"limitador-limitador-7c9b87d574-svfmk\" (UID: \"e8e6e17d-44a6-4e4e-bba4-236c02d5a856\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" Oct 30 14:16:37.284518 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.284489 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2nnw\" (UniqueName: \"kubernetes.io/projected/e8e6e17d-44a6-4e4e-bba4-236c02d5a856-kube-api-access-j2nnw\") pod \"limitador-limitador-7c9b87d574-svfmk\" (UID: \"e8e6e17d-44a6-4e4e-bba4-236c02d5a856\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" Oct 30 14:16:37.285133 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.285111 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e8e6e17d-44a6-4e4e-bba4-236c02d5a856-config-file\") pod \"limitador-limitador-7c9b87d574-svfmk\" (UID: \"e8e6e17d-44a6-4e4e-bba4-236c02d5a856\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" Oct 30 14:16:37.306456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.306423 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2nnw\" (UniqueName: \"kubernetes.io/projected/e8e6e17d-44a6-4e4e-bba4-236c02d5a856-kube-api-access-j2nnw\") pod \"limitador-limitador-7c9b87d574-svfmk\" (UID: \"e8e6e17d-44a6-4e4e-bba4-236c02d5a856\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" Oct 30 14:16:37.347051 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.347018 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" Oct 30 14:16:37.650208 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:37.650177 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:16:37.658193 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:16:37.658163 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e6e17d_44a6_4e4e_bba4_236c02d5a856.slice/crio-6aec8dee3fdf742136885c78b945cd6ce18a7a111ea26c1b1984ce184db952cf WatchSource:0}: Error finding container 6aec8dee3fdf742136885c78b945cd6ce18a7a111ea26c1b1984ce184db952cf: Status 404 returned error can't find the container with id 6aec8dee3fdf742136885c78b945cd6ce18a7a111ea26c1b1984ce184db952cf Oct 30 14:16:38.021766 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:38.021661 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" event={"ID":"e8e6e17d-44a6-4e4e-bba4-236c02d5a856","Type":"ContainerStarted","Data":"6aec8dee3fdf742136885c78b945cd6ce18a7a111ea26c1b1984ce184db952cf"} Oct 30 14:16:39.028309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:39.028220 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" event={"ID":"e8e6e17d-44a6-4e4e-bba4-236c02d5a856","Type":"ContainerStarted","Data":"fd47eb90a3620a1c51cfb272c2ca593bf415dab8e1eb5a4cb7d8338f793c4f0c"} Oct 30 14:16:39.028702 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:39.028361 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" Oct 30 14:16:39.067127 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:39.067073 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" podStartSLOduration=1.115269015 podStartE2EDuration="2.067059799s" totalImagesPullingTime="951.790784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:39.066467863 +0000 UTC m=+594.691911237" watchObservedRunningTime="2026-10-30 14:16:39.067059799 +0000 UTC m=+594.692503172" Oct 30 14:16:44.563042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:44.563008 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:16:44.567002 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:44.566980 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:16:50.034506 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:16:50.034474 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-svfmk" Oct 30 14:16:50.443537 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:16:50.443496 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:17:20.443067 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:17:20.443025 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:17:50.443305 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:17:50.443269 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:17:52.634475 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:17:52.634435 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:18:20.444100 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:18:20.444061 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:18:50.444604 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:18:50.444563 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:18:53.049050 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:53.049013 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:18:57.048639 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.048601 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:18:57.245836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.245800 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52"] Oct 30 14:18:57.249524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.249499 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.262836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.262809 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 14:18:57.272324 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.272301 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 14:18:57.272324 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.272300 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 14:18:57.272510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.272307 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-btpq5\"" Oct 30 14:18:57.274655 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.274632 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52"] Oct 30 14:18:57.344974 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.344942 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.344974 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.344975 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.345200 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.344999 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.345200 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.345146 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nv4\" (UniqueName: \"kubernetes.io/projected/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-kube-api-access-x5nv4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.345279 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.345199 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.345325 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.345298 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.445954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.445913 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.446142 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.445968 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.446142 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.445986 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.446142 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.446074 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.446329 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.446146 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nv4\" (UniqueName: \"kubernetes.io/projected/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-kube-api-access-x5nv4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.446329 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.446197 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.446504 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.446481 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.446777 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.446748 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.446920 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.446813 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.448525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.448506 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.448605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.448542 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.488794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.488763 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nv4\" (UniqueName: \"kubernetes.io/projected/bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8-kube-api-access-x5nv4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zlt52\" (UID: \"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.561817 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.561785 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:18:57.861108 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.860981 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52"] Oct 30 14:18:57.864749 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:18:57.864697 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbd0e4e_7de0_4a42_8cd4_c13fe2e90ae8.slice/crio-517ea954e3b4d51e8a15275cdc234690cb0a3033d9f400a78de327da62b43c22 WatchSource:0}: Error finding container 517ea954e3b4d51e8a15275cdc234690cb0a3033d9f400a78de327da62b43c22: Status 404 returned error can't find the container with id 517ea954e3b4d51e8a15275cdc234690cb0a3033d9f400a78de327da62b43c22 Oct 30 14:18:57.866077 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:57.866060 4968 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:18:58.638548 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:58.638507 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" event={"ID":"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8","Type":"ContainerStarted","Data":"517ea954e3b4d51e8a15275cdc234690cb0a3033d9f400a78de327da62b43c22"} Oct 30 14:18:59.537818 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.537783 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb"] Oct 30 14:18:59.541515 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.541489 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.565369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.565343 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.565504 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.565406 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4t7f\" (UniqueName: \"kubernetes.io/projected/5f2dd677-fdd5-452e-b283-740d207e3890-kube-api-access-q4t7f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.565667 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.565646 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.565755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.565664 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 14:18:59.565826 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.565758 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.565826 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.565785 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.566008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.565986 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f2dd677-fdd5-452e-b283-740d207e3890-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.575924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.575901 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb"] Oct 30 14:18:59.646114 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.646074 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" event={"ID":"bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8","Type":"ContainerStarted","Data":"5b448e695f1c236bb7a62f00d9b4a41e9fb524c5bfb8ce459eb816ebac62bec6"} Oct 30 14:18:59.667259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.667227 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f2dd677-fdd5-452e-b283-740d207e3890-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.667423 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.667264 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.667423 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.667299 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4t7f\" (UniqueName: \"kubernetes.io/projected/5f2dd677-fdd5-452e-b283-740d207e3890-kube-api-access-q4t7f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.667423 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.667347 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.667423 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.667374 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.667423 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.667398 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.667898 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.667876 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.667975 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.667950 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.668011 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.667976 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.669672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.669651 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f2dd677-fdd5-452e-b283-740d207e3890-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.669981 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.669963 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f2dd677-fdd5-452e-b283-740d207e3890-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.687311 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.687144 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" podStartSLOduration=1.6335507329999999 podStartE2EDuration="2.687127686s" totalImagesPullingTime="1.053576953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:59.685528975 +0000 UTC m=+735.310972349" watchObservedRunningTime="2026-10-30 14:18:59.687127686 +0000 UTC m=+735.312571058" Oct 30 14:18:59.689457 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.689431 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4t7f\" (UniqueName: \"kubernetes.io/projected/5f2dd677-fdd5-452e-b283-740d207e3890-kube-api-access-q4t7f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb\" (UID: \"5f2dd677-fdd5-452e-b283-740d207e3890\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:18:59.854768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:18:59.854729 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:19:00.155192 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:00.155099 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb"] Oct 30 14:19:00.159027 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:19:00.158996 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2dd677_fdd5_452e_b283_740d207e3890.slice/crio-4b90f609e7c3fec739fb706feb80e96aa6aef0750957263d10e02d01036d59e2 WatchSource:0}: Error finding container 4b90f609e7c3fec739fb706feb80e96aa6aef0750957263d10e02d01036d59e2: Status 404 returned error can't find the container with id 4b90f609e7c3fec739fb706feb80e96aa6aef0750957263d10e02d01036d59e2 Oct 30 14:19:00.562133 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:00.562102 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:19:00.653240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:00.653203 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" event={"ID":"5f2dd677-fdd5-452e-b283-740d207e3890","Type":"ContainerStarted","Data":"c59d0cbc3ff02a0d6905014ca5fcd081b21179efe045128ab1cbc8659ebdf381"} Oct 30 14:19:00.653614 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:00.653258 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" event={"ID":"5f2dd677-fdd5-452e-b283-740d207e3890","Type":"ContainerStarted","Data":"4b90f609e7c3fec739fb706feb80e96aa6aef0750957263d10e02d01036d59e2"} Oct 30 14:19:00.693124 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:00.693002 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" podStartSLOduration=1.485044314 podStartE2EDuration="1.692984508s" totalImagesPullingTime="207.940194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:00.691841211 +0000 UTC m=+736.317284584" watchObservedRunningTime="2026-10-30 14:19:00.692984508 +0000 UTC m=+736.318427882" Oct 30 14:19:00.855367 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:00.855332 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:19:01.165624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:01.165580 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:19:05.690476 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.690440 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg"] Oct 30 14:19:05.694917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.694890 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.717694 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.717659 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.717836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.717701 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.717836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.717791 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7sgq\" (UniqueName: \"kubernetes.io/projected/92142646-e68c-4054-b7bc-7fa5ac83b447-kube-api-access-w7sgq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.717836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.717820 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.717955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.717914 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.717955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.717945 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92142646-e68c-4054-b7bc-7fa5ac83b447-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.718809 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.718782 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 14:19:05.721461 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.721435 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg"] Oct 30 14:19:05.818610 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.818575 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92142646-e68c-4054-b7bc-7fa5ac83b447-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.818793 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.818617 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.818793 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.818635 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.818793 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.818685 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7sgq\" (UniqueName: \"kubernetes.io/projected/92142646-e68c-4054-b7bc-7fa5ac83b447-kube-api-access-w7sgq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.818793 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.818746 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.819002 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.818830 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.819122 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.819093 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.819122 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.819110 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.819230 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.819160 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.820980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.820954 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92142646-e68c-4054-b7bc-7fa5ac83b447-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.821138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.821119 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92142646-e68c-4054-b7bc-7fa5ac83b447-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:05.859918 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:05.859893 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7sgq\" (UniqueName: \"kubernetes.io/projected/92142646-e68c-4054-b7bc-7fa5ac83b447-kube-api-access-w7sgq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg\" (UID: \"92142646-e68c-4054-b7bc-7fa5ac83b447\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:06.010509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:06.010424 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:06.312246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:06.312065 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg"] Oct 30 14:19:06.317207 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:19:06.317175 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92142646_e68c_4054_b7bc_7fa5ac83b447.slice/crio-f7af7101b74dce1f808d7cd4965ecb9d3c3017e19fc303355f0aa73525d97a1e WatchSource:0}: Error finding container f7af7101b74dce1f808d7cd4965ecb9d3c3017e19fc303355f0aa73525d97a1e: Status 404 returned error can't find the container with id f7af7101b74dce1f808d7cd4965ecb9d3c3017e19fc303355f0aa73525d97a1e Oct 30 14:19:06.683019 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:06.682982 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" event={"ID":"92142646-e68c-4054-b7bc-7fa5ac83b447","Type":"ContainerStarted","Data":"0b9de3a09498202fa051330431c909963985e3be4a8d6ab57fc5f97478c86979"} Oct 30 14:19:06.683167 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:06.683043 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" event={"ID":"92142646-e68c-4054-b7bc-7fa5ac83b447","Type":"ContainerStarted","Data":"f7af7101b74dce1f808d7cd4965ecb9d3c3017e19fc303355f0aa73525d97a1e"} Oct 30 14:19:06.721879 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:06.721832 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" podStartSLOduration=1.482982202 podStartE2EDuration="1.721818226s" totalImagesPullingTime="238.836024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:06.721320579 +0000 UTC m=+742.346763952" watchObservedRunningTime="2026-10-30 14:19:06.721818226 +0000 UTC m=+742.347261599" Oct 30 14:19:07.011313 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:07.011274 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:07.562454 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:07.562423 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:19:07.575462 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:07.575439 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:19:07.663396 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:07.663362 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:19:07.700481 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:07.700448 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zlt52" Oct 30 14:19:09.855070 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:09.855023 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:19:09.867616 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:09.867592 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:19:10.714955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:10.714913 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb" Oct 30 14:19:16.011417 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:16.011378 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:16.023881 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:16.023856 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:16.742550 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:16.742524 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg" Oct 30 14:19:19.460753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.460704 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-6grbc"] Oct 30 14:19:19.464615 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.464592 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:19.478923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.478904 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 14:19:19.488666 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.488648 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 14:19:19.488768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.488650 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-qszds\"" Oct 30 14:19:19.490741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.490691 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-6grbc"] Oct 30 14:19:19.531845 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.531808 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5d8h\" (UniqueName: \"kubernetes.io/projected/470e0998-e4fc-4909-80ee-12a43c23c48f-kube-api-access-h5d8h\") pod \"ai-gateway-controller-7d6c696b49-6grbc\" (UID: \"470e0998-e4fc-4909-80ee-12a43c23c48f\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:19.531990 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.531852 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/470e0998-e4fc-4909-80ee-12a43c23c48f-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-6grbc\" (UID: \"470e0998-e4fc-4909-80ee-12a43c23c48f\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:19.633133 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.633097 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5d8h\" (UniqueName: \"kubernetes.io/projected/470e0998-e4fc-4909-80ee-12a43c23c48f-kube-api-access-h5d8h\") pod \"ai-gateway-controller-7d6c696b49-6grbc\" (UID: \"470e0998-e4fc-4909-80ee-12a43c23c48f\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:19.633300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.633145 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/470e0998-e4fc-4909-80ee-12a43c23c48f-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-6grbc\" (UID: \"470e0998-e4fc-4909-80ee-12a43c23c48f\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:19.635848 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.635706 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/470e0998-e4fc-4909-80ee-12a43c23c48f-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-6grbc\" (UID: \"470e0998-e4fc-4909-80ee-12a43c23c48f\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:19.654800 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.654769 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5d8h\" (UniqueName: \"kubernetes.io/projected/470e0998-e4fc-4909-80ee-12a43c23c48f-kube-api-access-h5d8h\") pod \"ai-gateway-controller-7d6c696b49-6grbc\" (UID: \"470e0998-e4fc-4909-80ee-12a43c23c48f\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:19.776909 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:19.776831 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:20.074627 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:20.074552 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-6grbc"] Oct 30 14:19:20.078174 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:19:20.078144 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470e0998_e4fc_4909_80ee_12a43c23c48f.slice/crio-834c28cde149d3a3117f044a4f223054a11dbe9977fe14115a10417de7d61edb WatchSource:0}: Error finding container 834c28cde149d3a3117f044a4f223054a11dbe9977fe14115a10417de7d61edb: Status 404 returned error can't find the container with id 834c28cde149d3a3117f044a4f223054a11dbe9977fe14115a10417de7d61edb Oct 30 14:19:20.443437 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:19:20.443393 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:19:20.752444 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:20.752352 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" event={"ID":"470e0998-e4fc-4909-80ee-12a43c23c48f","Type":"ContainerStarted","Data":"834c28cde149d3a3117f044a4f223054a11dbe9977fe14115a10417de7d61edb"} Oct 30 14:19:22.763745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:22.763633 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" event={"ID":"470e0998-e4fc-4909-80ee-12a43c23c48f","Type":"ContainerStarted","Data":"834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6"} Oct 30 14:19:22.764232 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:22.763768 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:22.802371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:22.802317 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" podStartSLOduration=1.433866736 podStartE2EDuration="3.802305928s" totalImagesPullingTime="2.368439192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:22.80109602 +0000 UTC m=+758.426539376" watchObservedRunningTime="2026-10-30 14:19:22.802305928 +0000 UTC m=+758.427749301" Oct 30 14:19:23.227703 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.227663 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4"] Oct 30 14:19:23.231516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.231494 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.256358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.256333 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sx525\"" Oct 30 14:19:23.257995 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.257975 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp"] Oct 30 14:19:23.261985 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.261950 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4"] Oct 30 14:19:23.262144 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.262068 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.272020 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.271993 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.272116 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.272044 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.272164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.272146 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qw64\" (UniqueName: \"kubernetes.io/projected/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-kube-api-access-5qw64\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.272216 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.272187 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.272304 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.272284 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.272363 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.272338 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.272438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.272409 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.272565 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.272487 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.272565 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.272510 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.284574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.284543 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp"] Oct 30 14:19:23.373053 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373020 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.373224 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373063 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.373224 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373096 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.373224 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373121 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.373224 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373149 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.373224 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373165 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.373485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373383 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.373485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373388 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.373572 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373495 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.373572 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373551 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.373658 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373566 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.373658 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373601 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.373658 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373641 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.373658 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373658 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.373878 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373677 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.373878 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373699 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.373878 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373742 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4wq\" (UniqueName: \"kubernetes.io/projected/43dc1f23-75ac-490a-9594-6ba78cb3d01e-kube-api-access-mv4wq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.373878 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373812 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.374110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373890 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.374110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373896 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qw64\" (UniqueName: \"kubernetes.io/projected/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-kube-api-access-5qw64\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.374110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373953 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.374110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.373968 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.374260 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.374116 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.375763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.375744 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.375923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.375909 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.407335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.407303 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.407579 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.407562 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qw64\" (UniqueName: \"kubernetes.io/projected/ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b-kube-api-access-5qw64\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zv6j4\" (UID: \"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.474919 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.474888 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.474926 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.474947 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.475071 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.475106 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.475146 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4wq\" (UniqueName: \"kubernetes.io/projected/43dc1f23-75ac-490a-9594-6ba78cb3d01e-kube-api-access-mv4wq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.475166 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.475242 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.475297 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475463 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.475385 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475463 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.475442 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475816 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.475788 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.475812 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.475917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.475858 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.477377 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.477353 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.477695 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.477678 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.498864 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.498807 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43dc1f23-75ac-490a-9594-6ba78cb3d01e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.499182 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.499165 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4wq\" (UniqueName: \"kubernetes.io/projected/43dc1f23-75ac-490a-9594-6ba78cb3d01e-kube-api-access-mv4wq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z2bwp\" (UID: \"43dc1f23-75ac-490a-9594-6ba78cb3d01e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.544461 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.544436 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:23.577338 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.577317 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:23.864851 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.864822 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4"] Oct 30 14:19:23.867143 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:19:23.867113 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf1f9b4_82da_4ad8_a5ee_bebce6df2b3b.slice/crio-da6e159b798fb994669d1c018b2d3a34618f656172850420c5e390b309b770c9 WatchSource:0}: Error finding container da6e159b798fb994669d1c018b2d3a34618f656172850420c5e390b309b770c9: Status 404 returned error can't find the container with id da6e159b798fb994669d1c018b2d3a34618f656172850420c5e390b309b770c9 Oct 30 14:19:23.912527 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:23.912451 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp"] Oct 30 14:19:23.914242 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:19:23.914218 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43dc1f23_75ac_490a_9594_6ba78cb3d01e.slice/crio-208284dc493b3a22a1eb4c8a2e1fa9bc0d65035f56177e9c54b692042843c29e WatchSource:0}: Error finding container 208284dc493b3a22a1eb4c8a2e1fa9bc0d65035f56177e9c54b692042843c29e: Status 404 returned error can't find the container with id 208284dc493b3a22a1eb4c8a2e1fa9bc0d65035f56177e9c54b692042843c29e Oct 30 14:19:24.776711 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:24.776671 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" event={"ID":"43dc1f23-75ac-490a-9594-6ba78cb3d01e","Type":"ContainerStarted","Data":"208284dc493b3a22a1eb4c8a2e1fa9bc0d65035f56177e9c54b692042843c29e"} Oct 30 14:19:24.778527 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:24.778494 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" event={"ID":"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b","Type":"ContainerStarted","Data":"da6e159b798fb994669d1c018b2d3a34618f656172850420c5e390b309b770c9"} Oct 30 14:19:25.132042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.132001 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6bfdff5577-j8ntq"] Oct 30 14:19:25.136250 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.136224 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" Oct 30 14:19:25.158129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.157952 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6bfdff5577-j8ntq"] Oct 30 14:19:25.187281 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.187246 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-t66vc\"" Oct 30 14:19:25.187465 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.187360 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 14:19:25.194701 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.194665 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bebba1-752e-47cc-b918-89c30bd62cf5-plugins-config-volume\") pod \"payload-processing-6bfdff5577-j8ntq\" (UID: \"76bebba1-752e-47cc-b918-89c30bd62cf5\") " pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" Oct 30 14:19:25.195010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.194806 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdhg\" (UniqueName: \"kubernetes.io/projected/76bebba1-752e-47cc-b918-89c30bd62cf5-kube-api-access-dzdhg\") pod \"payload-processing-6bfdff5577-j8ntq\" (UID: \"76bebba1-752e-47cc-b918-89c30bd62cf5\") " pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" Oct 30 14:19:25.296310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.296265 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdhg\" (UniqueName: \"kubernetes.io/projected/76bebba1-752e-47cc-b918-89c30bd62cf5-kube-api-access-dzdhg\") pod \"payload-processing-6bfdff5577-j8ntq\" (UID: \"76bebba1-752e-47cc-b918-89c30bd62cf5\") " pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" Oct 30 14:19:25.296537 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.296398 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bebba1-752e-47cc-b918-89c30bd62cf5-plugins-config-volume\") pod \"payload-processing-6bfdff5577-j8ntq\" (UID: \"76bebba1-752e-47cc-b918-89c30bd62cf5\") " pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" Oct 30 14:19:25.297129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.297093 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bebba1-752e-47cc-b918-89c30bd62cf5-plugins-config-volume\") pod \"payload-processing-6bfdff5577-j8ntq\" (UID: \"76bebba1-752e-47cc-b918-89c30bd62cf5\") " pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" Oct 30 14:19:25.326985 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.326949 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdhg\" (UniqueName: \"kubernetes.io/projected/76bebba1-752e-47cc-b918-89c30bd62cf5-kube-api-access-dzdhg\") pod \"payload-processing-6bfdff5577-j8ntq\" (UID: \"76bebba1-752e-47cc-b918-89c30bd62cf5\") " pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" Oct 30 14:19:25.453913 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.453820 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" Oct 30 14:19:25.777249 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:25.777100 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6bfdff5577-j8ntq"] Oct 30 14:19:26.799638 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:26.799601 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" event={"ID":"76bebba1-752e-47cc-b918-89c30bd62cf5","Type":"ContainerStarted","Data":"bd86a34e24b59dac66ec8f23a83d4f0303613013945864f5a2bf26bf7e395e24"} Oct 30 14:19:27.087899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:27.087849 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:19:27.088010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:27.087941 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:19:27.088010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:27.087984 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:19:27.223259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:27.222565 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:19:27.223259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:27.222656 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:19:27.223259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:27.222701 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:19:27.823310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:27.823202 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" event={"ID":"43dc1f23-75ac-490a-9594-6ba78cb3d01e","Type":"ContainerStarted","Data":"2de74fe7b4ccc732bb29d9342ded1472a18e6f0d754ea1797c8f593eb6583a70"} Oct 30 14:19:27.835766 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:27.835533 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" event={"ID":"ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b","Type":"ContainerStarted","Data":"b3d8ca47e9e07532ccef1d15c98c08c84595682e05fdaec4d073ac0817aab41c"} Oct 30 14:19:27.877286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:27.877228 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" podStartSLOduration=1.706593729 podStartE2EDuration="4.877209122s" totalImagesPullingTime="3.170615393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:27.876292603 +0000 UTC m=+763.501735973" watchObservedRunningTime="2026-10-30 14:19:27.877209122 +0000 UTC m=+763.502652501" Oct 30 14:19:27.938545 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:27.938470 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" podStartSLOduration=1.585917673 podStartE2EDuration="4.938449443s" totalImagesPullingTime="3.35253177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:27.935947763 +0000 UTC m=+763.561391148" watchObservedRunningTime="2026-10-30 14:19:27.938449443 +0000 UTC m=+763.563892819" Oct 30 14:19:28.544784 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:28.544756 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:28.546402 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:28.546377 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 14:19:28.546565 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:28.546548 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" podUID="ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 14:19:28.599847 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:28.578139 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:28.604752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:28.600908 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 14:19:28.604752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:28.600962 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" podUID="43dc1f23-75ac-490a-9594-6ba78cb3d01e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 14:19:28.842835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:28.842800 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" event={"ID":"76bebba1-752e-47cc-b918-89c30bd62cf5","Type":"ContainerStarted","Data":"76cffd45854cef8e9fd9b653f507deb1aa8dd130905df7e9b78c233d0dfa0088"} Oct 30 14:19:28.843252 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:28.842970 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" Oct 30 14:19:28.892122 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:28.892058 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" podStartSLOduration=1.120354095 podStartE2EDuration="3.892026185s" totalImagesPullingTime="2.77167209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:28.891193714 +0000 UTC m=+764.516637091" watchObservedRunningTime="2026-10-30 14:19:28.892026185 +0000 UTC m=+764.517469559" Oct 30 14:19:29.545106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:29.545075 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 14:19:29.545283 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:29.545126 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" podUID="ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 14:19:29.578049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:29.578018 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 14:19:29.578171 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:29.578073 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" podUID="43dc1f23-75ac-490a-9594-6ba78cb3d01e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 14:19:30.545597 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:30.545549 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 14:19:30.546009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:30.545637 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" podUID="ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 14:19:30.578320 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:30.578290 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 14:19:30.578441 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:30.578331 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" podUID="43dc1f23-75ac-490a-9594-6ba78cb3d01e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 14:19:31.545053 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:31.545012 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 14:19:31.545236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:31.545078 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" podUID="ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 14:19:31.578333 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:31.578300 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 14:19:31.578739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:31.578345 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" podUID="43dc1f23-75ac-490a-9594-6ba78cb3d01e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 14:19:32.545322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:32.545282 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 14:19:32.545500 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:32.545352 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" podUID="ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 14:19:32.578633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:32.578605 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 14:19:32.579094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:32.578648 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" podUID="43dc1f23-75ac-490a-9594-6ba78cb3d01e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 14:19:33.545612 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:33.545566 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 14:19:33.545836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:33.545635 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" podUID="ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 14:19:33.578494 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:33.578453 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 14:19:33.578672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:33.578513 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" podUID="43dc1f23-75ac-490a-9594-6ba78cb3d01e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 14:19:33.770743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:33.770690 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:34.544828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:34.544792 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 14:19:34.544997 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:34.544846 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" podUID="ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 14:19:34.578517 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:34.578483 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 14:19:34.578683 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:34.578538 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" podUID="43dc1f23-75ac-490a-9594-6ba78cb3d01e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 14:19:35.590161 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:35.590129 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:35.591155 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:35.591132 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:35.592045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:35.592020 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" Oct 30 14:19:35.696645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:35.693683 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:35.696645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:35.694611 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:35.696645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:35.695566 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" Oct 30 14:19:40.856025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:40.855994 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6bfdff5577-j8ntq" Oct 30 14:19:48.217546 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.217493 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-6grbc"] Oct 30 14:19:48.218002 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.217710 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" podUID="470e0998-e4fc-4909-80ee-12a43c23c48f" containerName="manager" containerID="cri-o://834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6" gracePeriod=30 Oct 30 14:19:48.476782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.476725 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:48.624791 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.624761 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470e0998-e4fc-4909-80ee-12a43c23c48f-kube-api-access-h5d8h\" (UniqueName: \"kubernetes.io/projected/470e0998-e4fc-4909-80ee-12a43c23c48f-kube-api-access-h5d8h\") pod \"470e0998-e4fc-4909-80ee-12a43c23c48f\" (UID: \"470e0998-e4fc-4909-80ee-12a43c23c48f\") " Oct 30 14:19:48.624960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.624844 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/470e0998-e4fc-4909-80ee-12a43c23c48f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/470e0998-e4fc-4909-80ee-12a43c23c48f-metrics-certs\") pod \"470e0998-e4fc-4909-80ee-12a43c23c48f\" (UID: \"470e0998-e4fc-4909-80ee-12a43c23c48f\") " Oct 30 14:19:48.627300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.627271 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470e0998-e4fc-4909-80ee-12a43c23c48f-metrics-certs" pod "470e0998-e4fc-4909-80ee-12a43c23c48f" (UID: "470e0998-e4fc-4909-80ee-12a43c23c48f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:19:48.627446 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.627348 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470e0998-e4fc-4909-80ee-12a43c23c48f-kube-api-access-h5d8h" pod "470e0998-e4fc-4909-80ee-12a43c23c48f" (UID: "470e0998-e4fc-4909-80ee-12a43c23c48f"). InnerVolumeSpecName "kube-api-access-h5d8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:19:48.725837 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.725809 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5d8h\" (UniqueName: \"kubernetes.io/projected/470e0998-e4fc-4909-80ee-12a43c23c48f-kube-api-access-h5d8h\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:19:48.725837 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.725833 4968 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/470e0998-e4fc-4909-80ee-12a43c23c48f-metrics-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:19:48.940955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.940923 4968 generic.go:356] "Generic (PLEG): container finished" podID="470e0998-e4fc-4909-80ee-12a43c23c48f" containerID="834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6" exitCode=0 Oct 30 14:19:48.941128 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.940971 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" Oct 30 14:19:48.941128 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.941010 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" event={"ID":"470e0998-e4fc-4909-80ee-12a43c23c48f","Type":"ContainerDied","Data":"834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6"} Oct 30 14:19:48.941128 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.941074 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-6grbc" event={"ID":"470e0998-e4fc-4909-80ee-12a43c23c48f","Type":"ContainerDied","Data":"834c28cde149d3a3117f044a4f223054a11dbe9977fe14115a10417de7d61edb"} Oct 30 14:19:48.941128 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.941100 4968 scope.go:122] "RemoveContainer" containerID="834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6" Oct 30 14:19:48.950449 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.950431 4968 scope.go:122] "RemoveContainer" containerID="834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6" Oct 30 14:19:48.950699 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:19:48.950681 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6\": container with ID starting with 834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6 not found: ID does not exist" containerID="834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6" Oct 30 14:19:48.950783 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.950710 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6"} err="failed to get container status \"834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6\": rpc error: code = NotFound desc = could not find container \"834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6\": container with ID starting with 834c0802a00a1c48a52848d1734d5d98abb2e20036c83cf4ff6cec66253cafd6 not found: ID does not exist" Oct 30 14:19:48.990956 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:48.990930 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-6grbc"] Oct 30 14:19:49.000706 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:49.000686 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-6grbc"] Oct 30 14:19:50.443562 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:19:50.443523 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:19:50.517371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:50.517337 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470e0998-e4fc-4909-80ee-12a43c23c48f" path="/var/lib/kubelet/pods/470e0998-e4fc-4909-80ee-12a43c23c48f/volumes" Oct 30 14:19:53.349883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.349847 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w"] Oct 30 14:19:53.350371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.350316 4968 cpu_manager.go:414] "containerMap: removing container" podUID="470e0998-e4fc-4909-80ee-12a43c23c48f" containerName="manager" Oct 30 14:19:53.350371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.350330 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470e0998-e4fc-4909-80ee-12a43c23c48f" containerName="manager" Oct 30 14:19:53.350456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.350405 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="470e0998-e4fc-4909-80ee-12a43c23c48f" containerName="manager" Oct 30 14:19:53.354987 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.354964 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.384706 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.384677 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w"] Oct 30 14:19:53.468950 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.468914 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.469132 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.469000 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.469132 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.469068 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8lsc\" (UniqueName: \"kubernetes.io/projected/3d721901-63ac-4136-a704-ae8283fce2bb-kube-api-access-t8lsc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.469132 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.469110 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.469132 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.469130 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.469317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.469149 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d721901-63ac-4136-a704-ae8283fce2bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.469374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.469324 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d721901-63ac-4136-a704-ae8283fce2bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.469417 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.469369 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d721901-63ac-4136-a704-ae8283fce2bb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.469456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.469442 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.570769 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.570676 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lsc\" (UniqueName: \"kubernetes.io/projected/3d721901-63ac-4136-a704-ae8283fce2bb-kube-api-access-t8lsc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571043 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.570838 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571043 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.570873 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571043 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.570902 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d721901-63ac-4136-a704-ae8283fce2bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571043 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.570994 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d721901-63ac-4136-a704-ae8283fce2bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.571072 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d721901-63ac-4136-a704-ae8283fce2bb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.571115 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.571134 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.571167 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571496 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.571369 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.571556 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571675 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.571566 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.571677 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.571834 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.571764 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d721901-63ac-4136-a704-ae8283fce2bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.573196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.573175 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d721901-63ac-4136-a704-ae8283fce2bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.573413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.573394 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d721901-63ac-4136-a704-ae8283fce2bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.593171 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.593149 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d721901-63ac-4136-a704-ae8283fce2bb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.593554 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.593535 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lsc\" (UniqueName: \"kubernetes.io/projected/3d721901-63ac-4136-a704-ae8283fce2bb-kube-api-access-t8lsc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zl46w\" (UID: \"3d721901-63ac-4136-a704-ae8283fce2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.670411 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.670318 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:53.987216 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.987137 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w"] Oct 30 14:19:53.990922 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:19:53.990898 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d721901_63ac_4136_a704_ae8283fce2bb.slice/crio-372046f612edda0bea26ed6acfe846ac7db449d27e4c65d9f1242925d23815af WatchSource:0}: Error finding container 372046f612edda0bea26ed6acfe846ac7db449d27e4c65d9f1242925d23815af: Status 404 returned error can't find the container with id 372046f612edda0bea26ed6acfe846ac7db449d27e4c65d9f1242925d23815af Oct 30 14:19:53.992870 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.992836 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:19:53.992952 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.992907 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:19:53.992952 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:53.992938 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:19:54.971960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:54.971922 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" event={"ID":"3d721901-63ac-4136-a704-ae8283fce2bb","Type":"ContainerStarted","Data":"5bd6d6a6fd52e15be7579b2f7302ed192733372e1accd55e09381db0f6d9b120"} Oct 30 14:19:54.972355 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:54.971979 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" event={"ID":"3d721901-63ac-4136-a704-ae8283fce2bb","Type":"ContainerStarted","Data":"372046f612edda0bea26ed6acfe846ac7db449d27e4c65d9f1242925d23815af"} Oct 30 14:19:55.050617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:55.050559 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" podStartSLOduration=2.050546249 podStartE2EDuration="2.050546249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:55.039008083 +0000 UTC m=+790.664451449" watchObservedRunningTime="2026-10-30 14:19:55.050546249 +0000 UTC m=+790.675989621" Oct 30 14:19:55.670745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:55.670700 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:19:55.672141 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:55.672119 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zl46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 14:19:55.672244 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:55.672193 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" podUID="3d721901-63ac-4136-a704-ae8283fce2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 14:19:56.671285 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:56.671250 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zl46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 14:19:56.671680 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:56.671304 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" podUID="3d721901-63ac-4136-a704-ae8283fce2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 14:19:57.670804 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:57.670772 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zl46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 14:19:57.671014 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:57.670830 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" podUID="3d721901-63ac-4136-a704-ae8283fce2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 14:19:58.671677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:58.671636 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zl46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 14:19:58.672124 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:58.671741 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" podUID="3d721901-63ac-4136-a704-ae8283fce2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 14:19:59.671296 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:59.671262 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zl46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 14:19:59.671466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:19:59.671327 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" podUID="3d721901-63ac-4136-a704-ae8283fce2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 14:20:00.670656 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:00.670617 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zl46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 14:20:00.671068 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:00.670675 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" podUID="3d721901-63ac-4136-a704-ae8283fce2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 14:20:02.179361 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:02.179319 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:20:02.180156 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:02.180136 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:20:02.180908 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:02.180887 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zl46w" Oct 30 14:20:15.242265 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.242226 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s"] Oct 30 14:20:15.252746 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.252706 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.278955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.278927 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a5ddf3-openshift-default-dockercfg-7z79k\"" Oct 30 14:20:15.281486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.281456 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s"] Oct 30 14:20:15.285464 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.285439 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-certs\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.285590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.285500 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-data\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.285657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.285633 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-credential-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.285758 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.285678 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b202666a-2b81-4d03-b286-a05d7a87b373-istio-podinfo\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.285758 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.285707 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.285843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.285771 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzvp\" (UniqueName: \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-kube-api-access-4jzvp\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.285843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.285820 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b202666a-2b81-4d03-b286-a05d7a87b373-istiod-ca-cert\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.285957 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.285937 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-envoy\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.286019 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.285996 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-istio-token\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.387186 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.387145 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-credential-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.387383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.387206 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b202666a-2b81-4d03-b286-a05d7a87b373-istio-podinfo\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.387383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.387251 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.387383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.387277 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzvp\" (UniqueName: \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-kube-api-access-4jzvp\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.387383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.387304 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b202666a-2b81-4d03-b286-a05d7a87b373-istiod-ca-cert\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.387383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.387334 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-envoy\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.387383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.387367 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-istio-token\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.387677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.387415 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-certs\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.387677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.387467 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-data\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.387816 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.387736 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.387873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.387837 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-data\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.388041 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.388021 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-certs\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.388145 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.388126 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b202666a-2b81-4d03-b286-a05d7a87b373-istiod-ca-cert\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.388233 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.388216 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-credential-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.390183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.390158 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-envoy\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.390746 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.390709 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b202666a-2b81-4d03-b286-a05d7a87b373-istio-podinfo\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.413438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.413407 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzvp\" (UniqueName: \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-kube-api-access-4jzvp\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.414460 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.414437 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-istio-token\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.566103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.566013 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:15.677392 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.677142 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65"] Oct 30 14:20:15.681617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.681593 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.711318 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.711291 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65"] Oct 30 14:20:15.755788 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.755754 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-137f01-openshift-default-dockercfg-wmd9z\"" Oct 30 14:20:15.790501 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.790459 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.790501 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.790501 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.790748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.790519 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtff\" (UniqueName: \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-kube-api-access-vrtff\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.790748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.790632 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.790748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.790656 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.790911 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.790756 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.790911 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.790801 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.790911 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.790863 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41493a9c-2413-46f7-ae3d-1c79505299b4-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.791060 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.790916 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.824081 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.823692 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh"] Oct 30 14:20:15.828146 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.828123 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.841648 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.841628 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-24dfd7-openshift-default-dockercfg-r2zgj\"" Oct 30 14:20:15.853555 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.853527 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh"] Oct 30 14:20:15.892017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.891986 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.892170 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892026 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.892170 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892081 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550b3932-faa4-4c46-9fa8-80c553a59dba-istiod-ca-cert\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.892170 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892111 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-data\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.892331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892309 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8cj7\" (UniqueName: \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-kube-api-access-c8cj7\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.892390 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892367 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.892444 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892406 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.892540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892523 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.892540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892520 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-token\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.892625 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892587 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.892672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892620 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-envoy\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.892705 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892673 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-certs\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.892705 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892684 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.892883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892768 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41493a9c-2413-46f7-ae3d-1c79505299b4-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.892883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892799 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-credential-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.892883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892845 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.893054 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892907 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.893054 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892948 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.893054 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892964 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtff\" (UniqueName: \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-kube-api-access-vrtff\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.893054 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.892981 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-podinfo\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.893416 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.893392 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41493a9c-2413-46f7-ae3d-1c79505299b4-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.893488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.893434 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.893488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.893449 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.896226 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.895789 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.896565 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.896541 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s"] Oct 30 14:20:15.896817 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.896787 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.900726 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:20:15.900693 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb202666a_2b81_4d03_b286_a05d7a87b373.slice/crio-fe3299ba242f3c8cbd8a5710bd294f8e86f34b0aa18a19b29f7ffb046b94bbc2 WatchSource:0}: Error finding container fe3299ba242f3c8cbd8a5710bd294f8e86f34b0aa18a19b29f7ffb046b94bbc2: Status 404 returned error can't find the container with id fe3299ba242f3c8cbd8a5710bd294f8e86f34b0aa18a19b29f7ffb046b94bbc2 Oct 30 14:20:15.902962 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.902928 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:15.903060 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.903003 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:15.903060 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.903048 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:15.918428 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.918399 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.918572 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.918554 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtff\" (UniqueName: \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-kube-api-access-vrtff\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:15.993793 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.993761 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.993926 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.993807 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-token\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.993926 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.993833 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-envoy\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.993926 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.993863 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-certs\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.993926 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.993904 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-credential-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.994252 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.993983 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-podinfo\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.994252 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.994216 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.994785 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.994342 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550b3932-faa4-4c46-9fa8-80c553a59dba-istiod-ca-cert\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.994785 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.994405 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-data\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.994785 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.994431 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8cj7\" (UniqueName: \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-kube-api-access-c8cj7\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.994785 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.994480 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-credential-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.994785 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.994512 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-certs\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.995099 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.995011 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-data\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.995151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.995129 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550b3932-faa4-4c46-9fa8-80c553a59dba-istiod-ca-cert\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.997370 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.997242 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-envoy\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:15.997549 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:15.997527 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-podinfo\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:16.003206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.002981 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:16.025970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.025914 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-token\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:16.026145 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.026120 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8cj7\" (UniqueName: \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-kube-api-access-c8cj7\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:16.107735 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.103108 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" event={"ID":"b202666a-2b81-4d03-b286-a05d7a87b373","Type":"ContainerStarted","Data":"24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9"} Oct 30 14:20:16.107735 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.103209 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" event={"ID":"b202666a-2b81-4d03-b286-a05d7a87b373","Type":"ContainerStarted","Data":"fe3299ba242f3c8cbd8a5710bd294f8e86f34b0aa18a19b29f7ffb046b94bbc2"} Oct 30 14:20:16.107735 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.104652 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf"] Oct 30 14:20:16.114974 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.112365 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.126371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.126344 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1d36cf-openshift-default-dockercfg-shmkc\"" Oct 30 14:20:16.134943 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.134914 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf"] Oct 30 14:20:16.143239 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.143059 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:16.152609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.152522 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" podStartSLOduration=1.152502991 podStartE2EDuration="1.152502991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:16.150872996 +0000 UTC m=+811.776316371" watchObservedRunningTime="2026-10-30 14:20:16.152502991 +0000 UTC m=+811.777946366" Oct 30 14:20:16.200660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.198564 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-data\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.200660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.198640 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istiod-ca-cert\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.200660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.198704 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.200660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.198761 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-credential-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.200660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.198859 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cjnc\" (UniqueName: \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-kube-api-access-5cjnc\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.200660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.198932 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-podinfo\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.200660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.198957 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-certs\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.200660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.198993 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-envoy\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.200660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.199025 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-token\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.300596 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.300533 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-data\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.301258 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.301232 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istiod-ca-cert\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.302359 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.301415 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.306257 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.302529 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-credential-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.306445 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.306423 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cjnc\" (UniqueName: \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-kube-api-access-5cjnc\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.306619 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.306602 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-podinfo\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.306758 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.306740 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-certs\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.306894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.306878 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-envoy\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.306999 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.306985 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-token\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.310612 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.310589 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-data\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.312359 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.311382 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istiod-ca-cert\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.313881 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.312783 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.314040 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.312999 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-certs\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.314266 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.313807 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-credential-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.316757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.316674 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-podinfo\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.317429 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.317408 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-envoy\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.336838 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.331473 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65"] Oct 30 14:20:16.344558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.344340 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cjnc\" (UniqueName: \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-kube-api-access-5cjnc\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.351939 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.347473 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:16.351939 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.347557 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:16.351939 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.347597 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:16.373132 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.373066 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-token\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.426964 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.426782 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:16.462582 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:20:16.462550 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550b3932_faa4_4c46_9fa8_80c553a59dba.slice/crio-1306177b77dd77d9f133b6707ec1e90e5c68cb73b80e4b4ed3a6f9b17867cd2e WatchSource:0}: Error finding container 1306177b77dd77d9f133b6707ec1e90e5c68cb73b80e4b4ed3a6f9b17867cd2e: Status 404 returned error can't find the container with id 1306177b77dd77d9f133b6707ec1e90e5c68cb73b80e4b4ed3a6f9b17867cd2e Oct 30 14:20:16.463146 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.463065 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh"] Oct 30 14:20:16.466635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.466593 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:16.466767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.466682 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:16.466833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.466767 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:16.566768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.566743 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:16.575322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.575293 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:16.757496 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.757347 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf"] Oct 30 14:20:16.760869 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:20:16.760841 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3caeb10_f0dd_4a43_988c_0f3b8ff8754e.slice/crio-984c52b083a7b1e83cf32c53f8487512ba4d637c01b00e48ff0777fe43244a87 WatchSource:0}: Error finding container 984c52b083a7b1e83cf32c53f8487512ba4d637c01b00e48ff0777fe43244a87: Status 404 returned error can't find the container with id 984c52b083a7b1e83cf32c53f8487512ba4d637c01b00e48ff0777fe43244a87 Oct 30 14:20:16.763885 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.763844 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:16.763987 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.763946 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:16.764041 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:16.763989 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:17.112513 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.112480 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" event={"ID":"550b3932-faa4-4c46-9fa8-80c553a59dba","Type":"ContainerStarted","Data":"867c0583c1961e04c65a0def98aa306bd82727eb2511f8204cdb8ffec7213c68"} Oct 30 14:20:17.112694 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.112536 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" event={"ID":"550b3932-faa4-4c46-9fa8-80c553a59dba","Type":"ContainerStarted","Data":"1306177b77dd77d9f133b6707ec1e90e5c68cb73b80e4b4ed3a6f9b17867cd2e"} Oct 30 14:20:17.114223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.114195 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" event={"ID":"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e","Type":"ContainerStarted","Data":"dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9"} Oct 30 14:20:17.114345 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.114243 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" event={"ID":"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e","Type":"ContainerStarted","Data":"984c52b083a7b1e83cf32c53f8487512ba4d637c01b00e48ff0777fe43244a87"} Oct 30 14:20:17.116032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.116001 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" event={"ID":"41493a9c-2413-46f7-ae3d-1c79505299b4","Type":"ContainerStarted","Data":"cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e"} Oct 30 14:20:17.116154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.116050 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" event={"ID":"41493a9c-2413-46f7-ae3d-1c79505299b4","Type":"ContainerStarted","Data":"4de968a8497a43b99aa6c38eb60926adcaa2024644e76735af79dfa89798d828"} Oct 30 14:20:17.116270 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.116253 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:17.117230 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.117208 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:17.144498 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.144476 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:17.149510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.149491 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:17.167513 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.167467 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" podStartSLOduration=2.167455717 podStartE2EDuration="2.167455717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:17.167099326 +0000 UTC m=+812.792542699" watchObservedRunningTime="2026-10-30 14:20:17.167455717 +0000 UTC m=+812.792899089" Oct 30 14:20:17.246544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.246491 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" podStartSLOduration=2.246479056 podStartE2EDuration="2.246479056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:17.24568134 +0000 UTC m=+812.871124713" watchObservedRunningTime="2026-10-30 14:20:17.246479056 +0000 UTC m=+812.871922428" Oct 30 14:20:17.311387 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.311315 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" podStartSLOduration=1.311300536 podStartE2EDuration="1.311300536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:17.308473976 +0000 UTC m=+812.933917348" watchObservedRunningTime="2026-10-30 14:20:17.311300536 +0000 UTC m=+812.936743908" Oct 30 14:20:17.427949 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.427877 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:17.433090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:17.433066 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:18.003508 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:18.003474 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:18.009205 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:18.009177 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:18.122950 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:18.122917 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:18.123542 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:18.123005 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:18.123542 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:18.123033 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:18.123841 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:18.123815 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:18.123963 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:18.123894 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:20:18.124071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:18.124058 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:20.444727 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:20:20.444683 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:20:20.634702 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.634662 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb"] Oct 30 14:20:20.640741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.639484 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:20:20.691338 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.691303 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb"] Oct 30 14:20:20.698265 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.698173 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-a5ddf3\"" Oct 30 14:20:20.698265 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.698263 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-a5ddf3-dockercfg-97z5z\"" Oct 30 14:20:20.772660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.772609 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb\" (UID: \"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:20:20.772862 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.772837 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9t67\" (UniqueName: \"kubernetes.io/projected/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-kube-api-access-m9t67\") pod \"payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb\" (UID: \"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:20:20.873406 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.873373 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb\" (UID: \"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:20:20.873579 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.873490 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9t67\" (UniqueName: \"kubernetes.io/projected/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-kube-api-access-m9t67\") pod \"payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb\" (UID: \"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:20:20.874157 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.874132 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb\" (UID: \"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:20:20.898954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.898929 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9t67\" (UniqueName: \"kubernetes.io/projected/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-kube-api-access-m9t67\") pod \"payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb\" (UID: \"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:20:20.960894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:20.960825 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:20:21.280911 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:21.280832 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb"] Oct 30 14:20:21.284510 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:20:21.284481 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2a455d_6c77_4a8f_a34f_5ce1de4c39e2.slice/crio-00b75d49544803b2ae49cbcfbc523a86c986fd8d8b98a632f0413e97dc39d5a3 WatchSource:0}: Error finding container 00b75d49544803b2ae49cbcfbc523a86c986fd8d8b98a632f0413e97dc39d5a3: Status 404 returned error can't find the container with id 00b75d49544803b2ae49cbcfbc523a86c986fd8d8b98a632f0413e97dc39d5a3 Oct 30 14:20:22.172736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.172605 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" event={"ID":"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2","Type":"ContainerStarted","Data":"287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a"} Oct 30 14:20:22.176412 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.173828 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" event={"ID":"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2","Type":"ContainerStarted","Data":"00b75d49544803b2ae49cbcfbc523a86c986fd8d8b98a632f0413e97dc39d5a3"} Oct 30 14:20:22.176412 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.173902 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:20:22.211058 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.211003 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" podStartSLOduration=2.210987964 podStartE2EDuration="2.210987964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:22.210696693 +0000 UTC m=+817.836140067" watchObservedRunningTime="2026-10-30 14:20:22.210987964 +0000 UTC m=+817.836431336" Oct 30 14:20:22.386193 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.386159 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4"] Oct 30 14:20:22.390166 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.390149 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:20:22.403520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.403501 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-7bf15f-dockercfg-j4thw\"" Oct 30 14:20:22.409099 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.409075 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4"] Oct 30 14:20:22.440361 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.440236 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-7bf15f\"" Oct 30 14:20:22.492328 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.492287 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg6xf\" (UniqueName: \"kubernetes.io/projected/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-kube-api-access-lg6xf\") pod \"payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4\" (UID: \"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:20:22.493618 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.493589 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4\" (UID: \"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:20:22.594929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.594895 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4\" (UID: \"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:20:22.595107 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.594941 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg6xf\" (UniqueName: \"kubernetes.io/projected/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-kube-api-access-lg6xf\") pod \"payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4\" (UID: \"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:20:22.595670 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.595647 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4\" (UID: \"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:20:22.616346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.616313 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg6xf\" (UniqueName: \"kubernetes.io/projected/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-kube-api-access-lg6xf\") pod \"payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4\" (UID: \"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:20:22.702905 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:22.702835 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:20:23.003581 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:23.003505 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4"] Oct 30 14:20:23.007124 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:20:23.007091 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe6f232_f757_4f7a_b60b_0fa7cb7a21f6.slice/crio-26d31ef935c2d9a37ab586e2b4819d3f84615ff2e164aaf63f074b917dc6291f WatchSource:0}: Error finding container 26d31ef935c2d9a37ab586e2b4819d3f84615ff2e164aaf63f074b917dc6291f: Status 404 returned error can't find the container with id 26d31ef935c2d9a37ab586e2b4819d3f84615ff2e164aaf63f074b917dc6291f Oct 30 14:20:23.179521 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:23.179469 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" event={"ID":"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6","Type":"ContainerStarted","Data":"c230a8f34fc1f286942a0e74048b0cd1341d3ec9c5e45972d919ca0b1efd7934"} Oct 30 14:20:23.180017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:23.179544 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" event={"ID":"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6","Type":"ContainerStarted","Data":"26d31ef935c2d9a37ab586e2b4819d3f84615ff2e164aaf63f074b917dc6291f"} Oct 30 14:20:23.180017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:23.179679 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:20:23.217321 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:23.217270 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" podStartSLOduration=1.217256242 podStartE2EDuration="1.217256242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:23.215638752 +0000 UTC m=+818.841082126" watchObservedRunningTime="2026-10-30 14:20:23.217256242 +0000 UTC m=+818.842699609" Oct 30 14:20:24.174618 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.174580 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9"] Oct 30 14:20:24.183742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.179378 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:20:24.200111 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.190184 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9"] Oct 30 14:20:24.208013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.207979 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-bf8266-dockercfg-898jw\"" Oct 30 14:20:24.217736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.212181 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-bf8266\"" Oct 30 14:20:24.321162 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.321130 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smhgk\" (UniqueName: \"kubernetes.io/projected/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-kube-api-access-smhgk\") pod \"payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9\" (UID: \"9e1775f1-3d4b-491c-b2dd-f207e5bd8025\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:20:24.321390 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.321272 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9\" (UID: \"9e1775f1-3d4b-491c-b2dd-f207e5bd8025\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:20:24.422504 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.422465 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9\" (UID: \"9e1775f1-3d4b-491c-b2dd-f207e5bd8025\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:20:24.422670 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.422603 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smhgk\" (UniqueName: \"kubernetes.io/projected/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-kube-api-access-smhgk\") pod \"payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9\" (UID: \"9e1775f1-3d4b-491c-b2dd-f207e5bd8025\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:20:24.423235 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.423216 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9\" (UID: \"9e1775f1-3d4b-491c-b2dd-f207e5bd8025\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:20:24.444285 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.444228 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smhgk\" (UniqueName: \"kubernetes.io/projected/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-kube-api-access-smhgk\") pod \"payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9\" (UID: \"9e1775f1-3d4b-491c-b2dd-f207e5bd8025\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:20:24.533471 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.533438 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:20:24.837208 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:20:24.837170 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1775f1_3d4b_491c_b2dd_f207e5bd8025.slice/crio-825abd52851be201e229e6bc3319c88b5c5ab7315fe15d14480839608493df81 WatchSource:0}: Error finding container 825abd52851be201e229e6bc3319c88b5c5ab7315fe15d14480839608493df81: Status 404 returned error can't find the container with id 825abd52851be201e229e6bc3319c88b5c5ab7315fe15d14480839608493df81 Oct 30 14:20:24.844975 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:24.844940 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9"] Oct 30 14:20:25.159186 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.159058 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw"] Oct 30 14:20:25.163922 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.163897 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:20:25.205371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.205344 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-24dfd7\"" Oct 30 14:20:25.205885 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.205524 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-24dfd7-dockercfg-n5fk7\"" Oct 30 14:20:25.207445 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.207418 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw"] Oct 30 14:20:25.225889 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.224769 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" event={"ID":"9e1775f1-3d4b-491c-b2dd-f207e5bd8025","Type":"ContainerStarted","Data":"2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4"} Oct 30 14:20:25.225889 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.224833 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" event={"ID":"9e1775f1-3d4b-491c-b2dd-f207e5bd8025","Type":"ContainerStarted","Data":"825abd52851be201e229e6bc3319c88b5c5ab7315fe15d14480839608493df81"} Oct 30 14:20:25.225889 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.225436 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:20:25.233316 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.233291 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dedc53-2e29-4eae-9ba3-267ae50634b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw\" (UID: \"13dedc53-2e29-4eae-9ba3-267ae50634b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:20:25.233475 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.233347 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhv4b\" (UniqueName: \"kubernetes.io/projected/13dedc53-2e29-4eae-9ba3-267ae50634b4-kube-api-access-bhv4b\") pod \"payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw\" (UID: \"13dedc53-2e29-4eae-9ba3-267ae50634b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:20:25.267248 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.267207 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" podStartSLOduration=1.2671944050000001 podStartE2EDuration="1.267194405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:25.267075333 +0000 UTC m=+820.892518707" watchObservedRunningTime="2026-10-30 14:20:25.267194405 +0000 UTC m=+820.892637778" Oct 30 14:20:25.334769 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.334707 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dedc53-2e29-4eae-9ba3-267ae50634b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw\" (UID: \"13dedc53-2e29-4eae-9ba3-267ae50634b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:20:25.334940 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.334805 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhv4b\" (UniqueName: \"kubernetes.io/projected/13dedc53-2e29-4eae-9ba3-267ae50634b4-kube-api-access-bhv4b\") pod \"payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw\" (UID: \"13dedc53-2e29-4eae-9ba3-267ae50634b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:20:25.335394 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.335373 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dedc53-2e29-4eae-9ba3-267ae50634b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw\" (UID: \"13dedc53-2e29-4eae-9ba3-267ae50634b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:20:25.356524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.356500 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhv4b\" (UniqueName: \"kubernetes.io/projected/13dedc53-2e29-4eae-9ba3-267ae50634b4-kube-api-access-bhv4b\") pod \"payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw\" (UID: \"13dedc53-2e29-4eae-9ba3-267ae50634b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:20:25.504415 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.504322 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:20:25.750128 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.750097 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:20:25.811402 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.811313 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw"] Oct 30 14:20:25.817691 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:20:25.817529 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dedc53_2e29_4eae_9ba3_267ae50634b4.slice/crio-9487136c04841be2d7292183fb04e9dfae881c89ecf638ab0707a1dee7d9d59b WatchSource:0}: Error finding container 9487136c04841be2d7292183fb04e9dfae881c89ecf638ab0707a1dee7d9d59b: Status 404 returned error can't find the container with id 9487136c04841be2d7292183fb04e9dfae881c89ecf638ab0707a1dee7d9d59b Oct 30 14:20:25.969440 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.969412 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247"] Oct 30 14:20:25.973499 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.973478 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:20:25.992171 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.992149 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-717995\"" Oct 30 14:20:25.992513 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.992495 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-717995-dockercfg-k6nnt\"" Oct 30 14:20:25.995308 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:25.995285 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247"] Oct 30 14:20:26.041635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.041602 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f24f5a6-883e-468e-bc08-4d9d47ec8864-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247\" (UID: \"8f24f5a6-883e-468e-bc08-4d9d47ec8864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:20:26.041841 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.041674 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8b94\" (UniqueName: \"kubernetes.io/projected/8f24f5a6-883e-468e-bc08-4d9d47ec8864-kube-api-access-c8b94\") pod \"payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247\" (UID: \"8f24f5a6-883e-468e-bc08-4d9d47ec8864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:20:26.142977 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.142880 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8b94\" (UniqueName: \"kubernetes.io/projected/8f24f5a6-883e-468e-bc08-4d9d47ec8864-kube-api-access-c8b94\") pod \"payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247\" (UID: \"8f24f5a6-883e-468e-bc08-4d9d47ec8864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:20:26.156395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.144966 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f24f5a6-883e-468e-bc08-4d9d47ec8864-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247\" (UID: \"8f24f5a6-883e-468e-bc08-4d9d47ec8864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:20:26.156395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.146045 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f24f5a6-883e-468e-bc08-4d9d47ec8864-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247\" (UID: \"8f24f5a6-883e-468e-bc08-4d9d47ec8864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:20:26.175755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.172629 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8b94\" (UniqueName: \"kubernetes.io/projected/8f24f5a6-883e-468e-bc08-4d9d47ec8864-kube-api-access-c8b94\") pod \"payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247\" (UID: \"8f24f5a6-883e-468e-bc08-4d9d47ec8864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:20:26.232686 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.232648 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" event={"ID":"13dedc53-2e29-4eae-9ba3-267ae50634b4","Type":"ContainerStarted","Data":"e83d84ec2ca8891666231b06789e9a3a7599ae9e95bd2ac0163c076201d6f979"} Oct 30 14:20:26.233209 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.232933 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:20:26.233209 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.232978 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" event={"ID":"13dedc53-2e29-4eae-9ba3-267ae50634b4","Type":"ContainerStarted","Data":"9487136c04841be2d7292183fb04e9dfae881c89ecf638ab0707a1dee7d9d59b"} Oct 30 14:20:26.277419 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.277363 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" podStartSLOduration=1.277350323 podStartE2EDuration="1.277350323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:26.27596909 +0000 UTC m=+821.901412464" watchObservedRunningTime="2026-10-30 14:20:26.277350323 +0000 UTC m=+821.902793696" Oct 30 14:20:26.286755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.286732 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:20:26.596707 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:26.596680 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247"] Oct 30 14:20:26.599831 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:20:26.599800 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f24f5a6_883e_468e_bc08_4d9d47ec8864.slice/crio-ec4bf08c7cf32ca34d21acf0e19a7f4157deaf0e2d8b21051f6e184e9c58b7ce WatchSource:0}: Error finding container ec4bf08c7cf32ca34d21acf0e19a7f4157deaf0e2d8b21051f6e184e9c58b7ce: Status 404 returned error can't find the container with id ec4bf08c7cf32ca34d21acf0e19a7f4157deaf0e2d8b21051f6e184e9c58b7ce Oct 30 14:20:27.240354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:27.240315 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" event={"ID":"8f24f5a6-883e-468e-bc08-4d9d47ec8864","Type":"ContainerStarted","Data":"b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a"} Oct 30 14:20:27.240873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:27.240497 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:20:27.240873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:27.240535 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" event={"ID":"8f24f5a6-883e-468e-bc08-4d9d47ec8864","Type":"ContainerStarted","Data":"ec4bf08c7cf32ca34d21acf0e19a7f4157deaf0e2d8b21051f6e184e9c58b7ce"} Oct 30 14:20:27.277884 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:27.277835 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" podStartSLOduration=2.277823286 podStartE2EDuration="2.277823286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:27.277446717 +0000 UTC m=+822.902890089" watchObservedRunningTime="2026-10-30 14:20:27.277823286 +0000 UTC m=+822.903266655" Oct 30 14:20:28.522593 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.522559 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb"] Oct 30 14:20:28.527266 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.527243 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:20:28.543927 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.543903 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1d36cf\"" Oct 30 14:20:28.547189 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.547166 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb"] Oct 30 14:20:28.570406 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.570369 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ea3629-deb5-427f-849b-2e40f1bbabc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb\" (UID: \"b3ea3629-deb5-427f-849b-2e40f1bbabc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:20:28.570564 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.570534 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqgh\" (UniqueName: \"kubernetes.io/projected/b3ea3629-deb5-427f-849b-2e40f1bbabc5-kube-api-access-6jqgh\") pod \"payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb\" (UID: \"b3ea3629-deb5-427f-849b-2e40f1bbabc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:20:28.604019 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.603965 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1d36cf-dockercfg-5l6rk\"" Oct 30 14:20:28.653741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.651340 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28"] Oct 30 14:20:28.657781 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.657753 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:20:28.661294 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.661266 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28"] Oct 30 14:20:28.671576 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.671548 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ea3629-deb5-427f-849b-2e40f1bbabc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb\" (UID: \"b3ea3629-deb5-427f-849b-2e40f1bbabc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:20:28.671846 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.671825 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqgh\" (UniqueName: \"kubernetes.io/projected/b3ea3629-deb5-427f-849b-2e40f1bbabc5-kube-api-access-6jqgh\") pod \"payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb\" (UID: \"b3ea3629-deb5-427f-849b-2e40f1bbabc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:20:28.672968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.672948 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ea3629-deb5-427f-849b-2e40f1bbabc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb\" (UID: \"b3ea3629-deb5-427f-849b-2e40f1bbabc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:20:28.694133 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.694106 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqgh\" (UniqueName: \"kubernetes.io/projected/b3ea3629-deb5-427f-849b-2e40f1bbabc5-kube-api-access-6jqgh\") pod \"payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb\" (UID: \"b3ea3629-deb5-427f-849b-2e40f1bbabc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:20:28.773177 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.773101 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0686340e-12ce-45b1-8135-2cf61f0a3f0b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28\" (UID: \"0686340e-12ce-45b1-8135-2cf61f0a3f0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:20:28.773294 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.773215 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h248c\" (UniqueName: \"kubernetes.io/projected/0686340e-12ce-45b1-8135-2cf61f0a3f0b-kube-api-access-h248c\") pod \"payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28\" (UID: \"0686340e-12ce-45b1-8135-2cf61f0a3f0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:20:28.840984 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.840945 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:20:28.874086 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.874053 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0686340e-12ce-45b1-8135-2cf61f0a3f0b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28\" (UID: \"0686340e-12ce-45b1-8135-2cf61f0a3f0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:20:28.874239 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.874220 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h248c\" (UniqueName: \"kubernetes.io/projected/0686340e-12ce-45b1-8135-2cf61f0a3f0b-kube-api-access-h248c\") pod \"payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28\" (UID: \"0686340e-12ce-45b1-8135-2cf61f0a3f0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:20:28.874954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.874929 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0686340e-12ce-45b1-8135-2cf61f0a3f0b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28\" (UID: \"0686340e-12ce-45b1-8135-2cf61f0a3f0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:20:28.896474 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.896444 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h248c\" (UniqueName: \"kubernetes.io/projected/0686340e-12ce-45b1-8135-2cf61f0a3f0b-kube-api-access-h248c\") pod \"payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28\" (UID: \"0686340e-12ce-45b1-8135-2cf61f0a3f0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:20:28.976359 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:28.976321 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:20:29.142868 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:29.142660 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb"] Oct 30 14:20:29.146871 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:20:29.145929 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ea3629_deb5_427f_849b_2e40f1bbabc5.slice/crio-33772b440407ed0777edc9e80b7dce0eb35b19b2feb08bed72d3c41f9fdf0c6c WatchSource:0}: Error finding container 33772b440407ed0777edc9e80b7dce0eb35b19b2feb08bed72d3c41f9fdf0c6c: Status 404 returned error can't find the container with id 33772b440407ed0777edc9e80b7dce0eb35b19b2feb08bed72d3c41f9fdf0c6c Oct 30 14:20:29.254382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:29.254341 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" event={"ID":"b3ea3629-deb5-427f-849b-2e40f1bbabc5","Type":"ContainerStarted","Data":"88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a"} Oct 30 14:20:29.254528 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:29.254416 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" event={"ID":"b3ea3629-deb5-427f-849b-2e40f1bbabc5","Type":"ContainerStarted","Data":"33772b440407ed0777edc9e80b7dce0eb35b19b2feb08bed72d3c41f9fdf0c6c"} Oct 30 14:20:29.254528 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:29.254448 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:20:29.284419 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:29.284344 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28"] Oct 30 14:20:29.288418 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:20:29.288382 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0686340e_12ce_45b1_8135_2cf61f0a3f0b.slice/crio-ac620769ef8307cedede84a4befc5718bff910447bce652b8eaa9a96ef164b44 WatchSource:0}: Error finding container ac620769ef8307cedede84a4befc5718bff910447bce652b8eaa9a96ef164b44: Status 404 returned error can't find the container with id ac620769ef8307cedede84a4befc5718bff910447bce652b8eaa9a96ef164b44 Oct 30 14:20:29.293101 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:29.293012 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" podStartSLOduration=1.2929949299999999 podStartE2EDuration="1.29299493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:29.292303473 +0000 UTC m=+824.917746848" watchObservedRunningTime="2026-10-30 14:20:29.29299493 +0000 UTC m=+824.918438305" Oct 30 14:20:29.563388 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:29.563317 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:20:30.262132 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:30.262095 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" event={"ID":"0686340e-12ce-45b1-8135-2cf61f0a3f0b","Type":"ContainerStarted","Data":"a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666"} Oct 30 14:20:30.262334 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:30.262150 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" event={"ID":"0686340e-12ce-45b1-8135-2cf61f0a3f0b","Type":"ContainerStarted","Data":"ac620769ef8307cedede84a4befc5718bff910447bce652b8eaa9a96ef164b44"} Oct 30 14:20:30.313326 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:30.313263 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" podStartSLOduration=2.313243233 podStartE2EDuration="2.313243233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:30.308542053 +0000 UTC m=+825.933985429" watchObservedRunningTime="2026-10-30 14:20:30.313243233 +0000 UTC m=+825.938686610" Oct 30 14:20:31.267688 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:31.267658 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:20:34.228571 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:34.228536 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:20:34.229087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:34.228622 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:20:35.210279 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:35.209849 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:20:37.243545 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:37.243519 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:20:37.244025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:37.243934 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:20:38.250343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:38.250212 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:20:40.265281 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:40.265251 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:20:41.150859 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:41.150824 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:20:41.391742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:41.391684 4968 generic.go:356] "Generic (PLEG): container finished" podID="b202666a-2b81-4d03-b286-a05d7a87b373" containerID="24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9" exitCode=0 Oct 30 14:20:41.392223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:41.391837 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" event={"ID":"b202666a-2b81-4d03-b286-a05d7a87b373","Type":"ContainerDied","Data":"24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9"} Oct 30 14:20:41.394749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:41.392290 4968 scope.go:122] "RemoveContainer" containerID="24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9" Oct 30 14:20:41.394749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:41.394512 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:41.394749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:41.394611 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:41.394749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:41.394662 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:42.275610 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:42.275582 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:20:42.421272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:42.421226 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" event={"ID":"b202666a-2b81-4d03-b286-a05d7a87b373","Type":"ContainerStarted","Data":"dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d"} Oct 30 14:20:42.566671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:42.566581 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:42.567789 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:42.567763 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 14:20:42.567913 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:42.567814 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 14:20:43.566346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:43.566313 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 14:20:43.566747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:43.566362 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 14:20:44.566699 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:44.566667 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 14:20:44.567120 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:44.566736 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 14:20:45.567090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:45.567032 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 14:20:45.567528 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:45.567102 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 14:20:46.566785 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:46.566744 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 14:20:46.566990 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:46.566795 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 14:20:47.117740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:47.116863 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:47.567232 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:47.567141 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 14:20:47.567232 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:47.567208 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 14:20:48.124272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:48.124239 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 14:20:48.124807 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:48.124289 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 14:20:48.475645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:48.475604 4968 generic.go:356] "Generic (PLEG): container finished" podID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerID="867c0583c1961e04c65a0def98aa306bd82727eb2511f8204cdb8ffec7213c68" exitCode=0 Oct 30 14:20:48.475818 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:48.475668 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" event={"ID":"550b3932-faa4-4c46-9fa8-80c553a59dba","Type":"ContainerDied","Data":"867c0583c1961e04c65a0def98aa306bd82727eb2511f8204cdb8ffec7213c68"} Oct 30 14:20:48.476100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:48.476087 4968 scope.go:122] "RemoveContainer" containerID="867c0583c1961e04c65a0def98aa306bd82727eb2511f8204cdb8ffec7213c68" Oct 30 14:20:48.477488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:48.477459 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:48.477548 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:48.477528 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:48.477582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:48.477561 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:48.572770 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:48.572747 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:48.574458 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:48.574436 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:20:49.503740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:49.501236 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" event={"ID":"550b3932-faa4-4c46-9fa8-80c553a59dba","Type":"ContainerStarted","Data":"c706d0ae0e30a21cd85d4030607cb29d1872df9d7aa7ca9a834d91f36709d023"} Oct 30 14:20:50.143986 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:50.143942 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:50.144171 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:50.144131 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 14:20:50.144214 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:50.144177 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 14:20:50.444349 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:20:50.444262 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:20:51.143914 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:51.143871 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 14:20:51.144426 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:51.143921 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 14:20:52.144234 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:52.143767 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 14:20:52.144234 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:52.143836 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 14:20:53.017737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:53.016807 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:20:53.144315 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:53.144269 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 14:20:53.144839 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:53.144345 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 14:20:54.144520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:54.144083 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 14:20:54.144520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:54.144153 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 14:20:54.562038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:54.561954 4968 generic.go:356] "Generic (PLEG): container finished" podID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerID="cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e" exitCode=0 Oct 30 14:20:54.562038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:54.562028 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" event={"ID":"41493a9c-2413-46f7-ae3d-1c79505299b4","Type":"ContainerDied","Data":"cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e"} Oct 30 14:20:54.562460 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:54.562437 4968 scope.go:122] "RemoveContainer" containerID="cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e" Oct 30 14:20:54.563828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:54.563799 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:54.563907 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:54.563873 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:54.563947 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:54.563905 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:20:55.148430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:55.148403 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:55.148904 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:55.148671 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:55.149447 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:55.149431 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:20:55.569331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:55.569247 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" event={"ID":"41493a9c-2413-46f7-ae3d-1c79505299b4","Type":"ContainerStarted","Data":"79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33"} Oct 30 14:20:56.006254 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:56.006164 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:20:56.006601 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:56.006578 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 14:20:56.006796 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:56.006768 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 14:20:57.007740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:57.007120 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 14:20:57.007740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:57.007187 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 14:20:58.004104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:58.004058 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 14:20:58.004314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:58.004109 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 14:20:59.004759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:59.004256 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 14:20:59.004759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:20:59.004321 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 14:21:00.013311 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:00.013269 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 14:21:00.013967 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:00.013930 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 14:21:01.086987 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:01.086550 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:21:01.087752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:01.087202 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:21:01.088062 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:01.088039 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:21:01.639992 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:01.639910 4968 generic.go:356] "Generic (PLEG): container finished" podID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerID="dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9" exitCode=0 Oct 30 14:21:01.640208 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:01.639984 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" event={"ID":"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e","Type":"ContainerDied","Data":"dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9"} Oct 30 14:21:01.640565 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:01.640546 4968 scope.go:122] "RemoveContainer" containerID="dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9" Oct 30 14:21:01.641787 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:01.641757 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:01.641857 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:01.641826 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:01.641894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:01.641864 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:02.204744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.193456 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6"] Oct 30 14:21:02.213221 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.213182 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.222447 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.222305 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6"] Oct 30 14:21:02.228902 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.228825 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-24dfd7\"/\"openshift-service-ca.crt\"" Oct 30 14:21:02.228902 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.228825 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-24dfd7\"/\"default-dockercfg-ggr84\"" Oct 30 14:21:02.229104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.228917 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-24dfd7\"/\"simulator-w3-24dfd7-kserve-self-signed-certs\"" Oct 30 14:21:02.236798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.236771 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-24dfd7\"/\"kube-root-ca.crt\"" Oct 30 14:21:02.277794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.277757 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tls-certs\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.277995 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.277811 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-model-cache\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.277995 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.277931 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tmp-dir\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.277995 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.277964 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-dshm\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.278167 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.278143 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpchv\" (UniqueName: \"kubernetes.io/projected/182fc12f-335a-4b5c-b26c-6ae65caa58cb-kube-api-access-lpchv\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.278217 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.278190 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-home\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.383379 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.383328 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpchv\" (UniqueName: \"kubernetes.io/projected/182fc12f-335a-4b5c-b26c-6ae65caa58cb-kube-api-access-lpchv\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.383664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.383640 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-home\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.384505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.383839 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tls-certs\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.384505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.383889 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-model-cache\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.384505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.383961 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tmp-dir\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.384505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.383994 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-dshm\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.384915 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.384886 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-home\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.385182 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.385157 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-model-cache\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.385424 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.385403 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tmp-dir\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.409786 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.398512 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tls-certs\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.410887 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.410850 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-dshm\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.413180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.413150 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpchv\" (UniqueName: \"kubernetes.io/projected/182fc12f-335a-4b5c-b26c-6ae65caa58cb-kube-api-access-lpchv\") pod \"simulator-w3-24dfd7-kserve-868784b9c7-m5bl6\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.534057 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.533974 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:02.652383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.650682 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" event={"ID":"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e","Type":"ContainerStarted","Data":"a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97"} Oct 30 14:21:02.903855 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:02.903811 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6"] Oct 30 14:21:02.907771 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:21:02.907737 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod182fc12f_335a_4b5c_b26c_6ae65caa58cb.slice/crio-bdad96c5e24cb6d2cf2910bab68e3eb957a7c3245bfe637ecf991aa455f45aa7 WatchSource:0}: Error finding container bdad96c5e24cb6d2cf2910bab68e3eb957a7c3245bfe637ecf991aa455f45aa7: Status 404 returned error can't find the container with id bdad96c5e24cb6d2cf2910bab68e3eb957a7c3245bfe637ecf991aa455f45aa7 Oct 30 14:21:03.122889 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:03.122855 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:21:03.427451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:03.427418 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:21:03.428503 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:03.428476 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 14:21:03.428559 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:03.428536 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 14:21:03.658554 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:03.658508 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" event={"ID":"182fc12f-335a-4b5c-b26c-6ae65caa58cb","Type":"ContainerStarted","Data":"f7535bf5b4b32715b40a1cab9587e8c5a14b23569a05daa901ddcaf8ce9b03eb"} Oct 30 14:21:03.658554 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:03.658552 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" event={"ID":"182fc12f-335a-4b5c-b26c-6ae65caa58cb","Type":"ContainerStarted","Data":"bdad96c5e24cb6d2cf2910bab68e3eb957a7c3245bfe637ecf991aa455f45aa7"} Oct 30 14:21:03.699911 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:03.699793 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" podStartSLOduration=1.699774557 podStartE2EDuration="1.699774557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:03.69917473 +0000 UTC m=+859.324618104" watchObservedRunningTime="2026-10-30 14:21:03.699774557 +0000 UTC m=+859.325217933" Oct 30 14:21:04.427189 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:04.427155 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 14:21:04.427367 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:04.427209 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 14:21:04.534494 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:04.534461 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:05.427220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:05.427177 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 14:21:05.427427 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:05.427240 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 14:21:05.800798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:05.795540 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:21:06.427832 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:06.427790 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 14:21:06.428013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:06.427860 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 14:21:07.427218 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:07.427184 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 14:21:07.427616 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:07.427241 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 14:21:08.431208 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:08.431175 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:21:08.432261 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:08.432238 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:21:12.534956 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:12.534924 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:12.547928 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:12.547901 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:12.771007 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:12.770979 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:21:19.203141 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:19.203105 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:21:20.444065 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:21:20.444028 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:21:28.003315 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.003281 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks"] Oct 30 14:21:28.008414 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.008387 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.034283 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.034252 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-7bf15f\"/\"kube-root-ca.crt\"" Oct 30 14:21:28.034430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.034337 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-7bf15f\"/\"openshift-service-ca.crt\"" Oct 30 14:21:28.034430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.034410 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7bf15f\"/\"premium-w5-7bf15f-kserve-self-signed-certs\"" Oct 30 14:21:28.034522 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.034505 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7bf15f\"/\"default-dockercfg-br9d6\"" Oct 30 14:21:28.037348 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.037327 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks"] Oct 30 14:21:28.138509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.138470 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2558\" (UniqueName: \"kubernetes.io/projected/aca3c382-c905-4be5-ae1a-d5f9336adc1d-kube-api-access-l2558\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.138709 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.138522 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-dshm\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.138709 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.138699 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tmp-dir\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.138861 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.138777 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-home\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.138861 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.138817 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tls-certs\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.138979 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.138886 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-model-cache\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.240740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.240296 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2558\" (UniqueName: \"kubernetes.io/projected/aca3c382-c905-4be5-ae1a-d5f9336adc1d-kube-api-access-l2558\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.240740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.240354 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-dshm\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.240740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.240442 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tmp-dir\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.240740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.240490 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-home\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.240740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.240513 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tls-certs\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.240740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.240586 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-model-cache\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.242281 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.241938 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tmp-dir\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.242683 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.242662 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-home\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.243081 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.243062 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-model-cache\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.252302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.250425 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tls-certs\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.252302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.252214 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-dshm\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.265339 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.265239 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2558\" (UniqueName: \"kubernetes.io/projected/aca3c382-c905-4be5-ae1a-d5f9336adc1d-kube-api-access-l2558\") pod \"premium-w5-7bf15f-kserve-68576f66d-vl8ks\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.321504 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.321471 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:28.631233 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.631203 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks"] Oct 30 14:21:28.634427 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:21:28.634400 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaca3c382_c905_4be5_ae1a_d5f9336adc1d.slice/crio-2e1ede62e097bfc00ca12f8a08991751ac77863402c630baa9cf43f1f0a34d9f WatchSource:0}: Error finding container 2e1ede62e097bfc00ca12f8a08991751ac77863402c630baa9cf43f1f0a34d9f: Status 404 returned error can't find the container with id 2e1ede62e097bfc00ca12f8a08991751ac77863402c630baa9cf43f1f0a34d9f Oct 30 14:21:28.858321 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.858283 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" event={"ID":"aca3c382-c905-4be5-ae1a-d5f9336adc1d","Type":"ContainerStarted","Data":"4a4ece1581879a3f8fe49c45f1de7155206b1d1d099a5294aad8502d4eda042e"} Oct 30 14:21:28.858321 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.858327 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" event={"ID":"aca3c382-c905-4be5-ae1a-d5f9336adc1d","Type":"ContainerStarted","Data":"2e1ede62e097bfc00ca12f8a08991751ac77863402c630baa9cf43f1f0a34d9f"} Oct 30 14:21:28.906143 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:28.906036 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" podStartSLOduration=1.906020996 podStartE2EDuration="1.906020996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:28.903565158 +0000 UTC m=+884.529008531" watchObservedRunningTime="2026-10-30 14:21:28.906020996 +0000 UTC m=+884.531464370" Oct 30 14:21:29.322607 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:29.322522 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:30.921074 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.921036 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl"] Oct 30 14:21:30.924259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.924241 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:30.940192 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.940172 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-7bf15f-openshift-default-dockercfg-d4kcn\"" Oct 30 14:21:30.977985 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.977957 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:30.978112 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.978001 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvgsk\" (UniqueName: \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-kube-api-access-xvgsk\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:30.978112 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.978067 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:30.978278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.978254 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:30.978343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.978324 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:30.978402 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.978369 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:30.978402 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.978391 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:30.978494 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.978447 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:30.978553 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.978528 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:30.979428 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:30.979403 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl"] Oct 30 14:21:31.079483 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.079446 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.079652 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.079520 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.079652 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.079551 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.079765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.079681 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.079765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.079757 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.079849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.079786 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.079849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.079819 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.079970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.079886 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.079970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.079914 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvgsk\" (UniqueName: \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-kube-api-access-xvgsk\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.080160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.080135 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.080270 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.080230 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.080353 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.080330 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.080439 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.080424 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.080480 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.080452 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.082045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.082022 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.082045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.082040 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.101828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.101802 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.101950 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.101886 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvgsk\" (UniqueName: \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-kube-api-access-xvgsk\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.238191 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.238110 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:31.565274 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.565199 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl"] Oct 30 14:21:31.568642 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:21:31.568616 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0ef2f1_6d69_45c0_bfe1_60438eeefb03.slice/crio-8070089dc2bbf2cc2d0210b69439180e511e5b73ba721ca66fc756269b7114a4 WatchSource:0}: Error finding container 8070089dc2bbf2cc2d0210b69439180e511e5b73ba721ca66fc756269b7114a4: Status 404 returned error can't find the container with id 8070089dc2bbf2cc2d0210b69439180e511e5b73ba721ca66fc756269b7114a4 Oct 30 14:21:31.571466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.571434 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:31.571549 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.571499 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:31.571549 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.571529 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:31.880665 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.880630 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" event={"ID":"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03","Type":"ContainerStarted","Data":"50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606"} Oct 30 14:21:31.880889 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.880694 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" event={"ID":"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03","Type":"ContainerStarted","Data":"8070089dc2bbf2cc2d0210b69439180e511e5b73ba721ca66fc756269b7114a4"} Oct 30 14:21:31.926922 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:31.926859 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" podStartSLOduration=1.926838145 podStartE2EDuration="1.926838145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:31.924514367 +0000 UTC m=+887.549957741" watchObservedRunningTime="2026-10-30 14:21:31.926838145 +0000 UTC m=+887.552281519" Oct 30 14:21:32.238945 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:32.238859 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:32.240465 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:32.240437 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:21:32.240618 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:32.240493 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:21:33.238649 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:33.238610 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:21:33.239075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:33.238677 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:21:34.238901 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:34.238865 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:21:34.239295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:34.238933 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:21:35.008834 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:35.008790 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:21:35.238980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:35.238937 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:21:35.239482 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:35.238999 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:21:36.238860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:36.238824 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:21:36.239032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:36.238894 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:21:37.239257 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:37.239211 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:21:37.239784 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:37.239275 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:21:38.322539 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:38.322500 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:38.337381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:38.337348 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:38.925914 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:38.925879 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:38.926747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:38.926709 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:38.928231 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:38.928205 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:21:38.965660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:38.965637 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:21:41.883750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.883680 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q"] Oct 30 14:21:41.889627 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.889599 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:41.903800 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.903766 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a5ddf3\"/\"kube-root-ca.crt\"" Oct 30 14:21:41.906113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.906088 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a5ddf3\"/\"openshift-service-ca.crt\"" Oct 30 14:21:41.916332 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.916155 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a5ddf3\"/\"premium-w2-a5ddf3-kserve-self-signed-certs\"" Oct 30 14:21:41.916654 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.916633 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a5ddf3\"/\"default-dockercfg-8cscx\"" Oct 30 14:21:41.919829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.919802 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q"] Oct 30 14:21:41.950960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.950935 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-tmp-dir\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:41.951100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.950986 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-model-cache\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:41.951100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.951044 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-home\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:41.951100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.951074 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-dshm\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:41.951207 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.951139 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b00b7d-0f8b-41f7-b646-995797d34045-tls-certs\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:41.951207 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:41.951180 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwkl7\" (UniqueName: \"kubernetes.io/projected/99b00b7d-0f8b-41f7-b646-995797d34045-kube-api-access-xwkl7\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.052527 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.052491 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-model-cache\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.052527 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.052532 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-home\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.052738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.052664 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-dshm\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.052782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.052768 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b00b7d-0f8b-41f7-b646-995797d34045-tls-certs\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.052863 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.052845 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwkl7\" (UniqueName: \"kubernetes.io/projected/99b00b7d-0f8b-41f7-b646-995797d34045-kube-api-access-xwkl7\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.052948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.052880 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-home\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.052948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.052931 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-model-cache\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.053052 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.052965 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-tmp-dir\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.053257 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.053235 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-tmp-dir\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.054904 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.054853 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-dshm\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.055626 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.055602 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b00b7d-0f8b-41f7-b646-995797d34045-tls-certs\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.074794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.074756 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwkl7\" (UniqueName: \"kubernetes.io/projected/99b00b7d-0f8b-41f7-b646-995797d34045-kube-api-access-xwkl7\") pod \"premium-w2-a5ddf3-kserve-f5765d85f-j4x4q\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.203913 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.203826 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:42.507247 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.507161 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q"] Oct 30 14:21:42.511228 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:21:42.511192 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99b00b7d_0f8b_41f7_b646_995797d34045.slice/crio-44cfd24fcc9749faa4ee3b148c0d85877fe0c8187f32850391e057252330c584 WatchSource:0}: Error finding container 44cfd24fcc9749faa4ee3b148c0d85877fe0c8187f32850391e057252330c584: Status 404 returned error can't find the container with id 44cfd24fcc9749faa4ee3b148c0d85877fe0c8187f32850391e057252330c584 Oct 30 14:21:42.981743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.981676 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" event={"ID":"99b00b7d-0f8b-41f7-b646-995797d34045","Type":"ContainerStarted","Data":"531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75"} Oct 30 14:21:42.981743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:42.981734 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" event={"ID":"99b00b7d-0f8b-41f7-b646-995797d34045","Type":"ContainerStarted","Data":"44cfd24fcc9749faa4ee3b148c0d85877fe0c8187f32850391e057252330c584"} Oct 30 14:21:43.022042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:43.021981 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" podStartSLOduration=2.021964818 podStartE2EDuration="2.021964818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:43.019291892 +0000 UTC m=+898.644735264" watchObservedRunningTime="2026-10-30 14:21:43.021964818 +0000 UTC m=+898.647408185" Oct 30 14:21:43.204599 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:43.204569 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:44.608101 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:44.608073 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:21:44.615079 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:44.615057 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:21:46.574408 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:46.574379 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:21:50.445373 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:21:50.445334 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:21:52.204484 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.204445 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:52.217511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.217482 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:52.773557 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.773518 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj"] Oct 30 14:21:52.789305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.789270 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.807629 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.807599 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-137f01\"/\"openshift-service-ca.crt\"" Oct 30 14:21:52.807794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.807646 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-137f01\"/\"kube-root-ca.crt\"" Oct 30 14:21:52.809703 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.809679 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj"] Oct 30 14:21:52.812560 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.812538 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-137f01\"/\"default-dockercfg-5n5lk\"" Oct 30 14:21:52.812849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.812831 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-137f01\"/\"premium-w4-137f01-kserve-self-signed-certs\"" Oct 30 14:21:52.882747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.882696 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-dshm\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.882923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.882818 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-home\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.882923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.882879 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tmp-dir\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.882923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.882914 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-model-cache\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.883057 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.883024 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tls-certs\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.883133 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.883119 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wzlg\" (UniqueName: \"kubernetes.io/projected/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-kube-api-access-9wzlg\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.984025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.983974 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-model-cache\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.986195 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.986168 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tls-certs\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.987171 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.987147 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wzlg\" (UniqueName: \"kubernetes.io/projected/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-kube-api-access-9wzlg\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.987666 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.987646 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-dshm\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.993754 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.987848 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-home\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.993983 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.993962 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tmp-dir\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.994519 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.990282 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-home\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.994628 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.994379 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tmp-dir\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.994628 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.984848 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-model-cache\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:52.997990 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:52.997967 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-dshm\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:53.006974 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:53.006945 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tls-certs\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:53.012736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:53.009998 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wzlg\" (UniqueName: \"kubernetes.io/projected/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-kube-api-access-9wzlg\") pod \"premium-w4-137f01-kserve-65c5f9cd5b-r5tmj\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:53.061341 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:53.061265 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:21:53.104301 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:53.104268 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:53.419407 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:53.419013 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj"] Oct 30 14:21:53.423062 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:21:53.423016 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1dd55df_e0b0_4ffe_a221_96fd8ff4b3fa.slice/crio-ea7ee8d61bf50d12d5fbe3f10617880607e0b624b46003610c8e6476501326a4 WatchSource:0}: Error finding container ea7ee8d61bf50d12d5fbe3f10617880607e0b624b46003610c8e6476501326a4: Status 404 returned error can't find the container with id ea7ee8d61bf50d12d5fbe3f10617880607e0b624b46003610c8e6476501326a4 Oct 30 14:21:54.057100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:54.057062 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" event={"ID":"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa","Type":"ContainerStarted","Data":"b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201"} Oct 30 14:21:54.057100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:54.057105 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" event={"ID":"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa","Type":"ContainerStarted","Data":"ea7ee8d61bf50d12d5fbe3f10617880607e0b624b46003610c8e6476501326a4"} Oct 30 14:21:54.104663 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:54.104623 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:21:54.117853 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:21:54.117797 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" podStartSLOduration=2.117779001 podStartE2EDuration="2.117779001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:54.115097778 +0000 UTC m=+909.740541156" watchObservedRunningTime="2026-10-30 14:21:54.117779001 +0000 UTC m=+909.743222375" Oct 30 14:22:03.104877 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:03.104840 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:22:03.118439 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:03.118408 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:22:04.117653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:04.117623 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:22:06.162371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:06.162341 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:22:12.964913 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:12.964871 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5"] Oct 30 14:22:12.969585 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:12.969566 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:12.994317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:12.994291 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:22:12.994600 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:12.994577 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:22:12.994744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:12.994689 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:22:12.994821 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:12.994761 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:22:12.997226 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:12.997205 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5"] Oct 30 14:22:13.088753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:13.088697 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxpjm\" (UniqueName: \"kubernetes.io/projected/e5a502d9-143b-450e-9096-fc733d804245-kube-api-access-fxpjm\") pod \"maas-api-revoke-sub-8b8212e5-tpbc5\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:13.088948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:13.088924 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a502d9-143b-450e-9096-fc733d804245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b8212e5-tpbc5\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:13.089015 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:13.088970 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a502d9-143b-450e-9096-fc733d804245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b8212e5-tpbc5\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:13.190520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:13.190475 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a502d9-143b-450e-9096-fc733d804245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b8212e5-tpbc5\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:13.190705 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:13.190537 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a502d9-143b-450e-9096-fc733d804245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b8212e5-tpbc5\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:13.190705 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:13.190575 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxpjm\" (UniqueName: \"kubernetes.io/projected/e5a502d9-143b-450e-9096-fc733d804245-kube-api-access-fxpjm\") pod \"maas-api-revoke-sub-8b8212e5-tpbc5\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:13.191450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:13.191427 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a502d9-143b-450e-9096-fc733d804245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b8212e5-tpbc5\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:13.192955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:13.192928 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a502d9-143b-450e-9096-fc733d804245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b8212e5-tpbc5\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:13.212598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:13.212572 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxpjm\" (UniqueName: \"kubernetes.io/projected/e5a502d9-143b-450e-9096-fc733d804245-kube-api-access-fxpjm\") pod \"maas-api-revoke-sub-8b8212e5-tpbc5\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:13.283622 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:13.283554 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:13.593129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:13.593089 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5"] Oct 30 14:22:13.596951 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:13.596918 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a502d9_143b_450e_9096_fc733d804245.slice/crio-086949254fc3006849ed29056101c77418b572cfaea251f69a9ca92b37ac9287 WatchSource:0}: Error finding container 086949254fc3006849ed29056101c77418b572cfaea251f69a9ca92b37ac9287: Status 404 returned error can't find the container with id 086949254fc3006849ed29056101c77418b572cfaea251f69a9ca92b37ac9287 Oct 30 14:22:14.194661 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:14.194628 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" event={"ID":"e5a502d9-143b-450e-9096-fc733d804245","Type":"ContainerStarted","Data":"086949254fc3006849ed29056101c77418b572cfaea251f69a9ca92b37ac9287"} Oct 30 14:22:17.278120 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:17.278076 4968 generic.go:356] "Generic (PLEG): container finished" podID="e5a502d9-143b-450e-9096-fc733d804245" containerID="b319d5cab1930070eacd0343e808ad1d9232b0cc444e1df42a165d10b7e1b2a4" exitCode=0 Oct 30 14:22:17.278627 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:17.278129 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" event={"ID":"e5a502d9-143b-450e-9096-fc733d804245","Type":"ContainerDied","Data":"b319d5cab1930070eacd0343e808ad1d9232b0cc444e1df42a165d10b7e1b2a4"} Oct 30 14:22:18.434782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:18.434753 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:18.471953 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:18.471918 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5a502d9-143b-450e-9096-fc733d804245-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a502d9-143b-450e-9096-fc733d804245-maas-api-xks-ca\") pod \"e5a502d9-143b-450e-9096-fc733d804245\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " Oct 30 14:22:18.472111 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:18.471970 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a502d9-143b-450e-9096-fc733d804245-kube-api-access-fxpjm\" (UniqueName: \"kubernetes.io/projected/e5a502d9-143b-450e-9096-fc733d804245-kube-api-access-fxpjm\") pod \"e5a502d9-143b-450e-9096-fc733d804245\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " Oct 30 14:22:18.472197 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:18.472182 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a502d9-143b-450e-9096-fc733d804245-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a502d9-143b-450e-9096-fc733d804245-maas-api-openshift-ca\") pod \"e5a502d9-143b-450e-9096-fc733d804245\" (UID: \"e5a502d9-143b-450e-9096-fc733d804245\") " Oct 30 14:22:18.472611 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:18.472581 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a502d9-143b-450e-9096-fc733d804245-maas-api-openshift-ca" pod "e5a502d9-143b-450e-9096-fc733d804245" (UID: "e5a502d9-143b-450e-9096-fc733d804245"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:18.474432 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:18.474407 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a502d9-143b-450e-9096-fc733d804245-kube-api-access-fxpjm" pod "e5a502d9-143b-450e-9096-fc733d804245" (UID: "e5a502d9-143b-450e-9096-fc733d804245"). InnerVolumeSpecName "kube-api-access-fxpjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:18.474530 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:18.474487 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a502d9-143b-450e-9096-fc733d804245-maas-api-xks-ca" pod "e5a502d9-143b-450e-9096-fc733d804245" (UID: "e5a502d9-143b-450e-9096-fc733d804245"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:18.574495 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:18.574289 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a502d9-143b-450e-9096-fc733d804245-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:18.574495 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:18.574403 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxpjm\" (UniqueName: \"kubernetes.io/projected/e5a502d9-143b-450e-9096-fc733d804245-kube-api-access-fxpjm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:18.574495 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:18.574437 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a502d9-143b-450e-9096-fc733d804245-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:19.293128 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.293082 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" event={"ID":"e5a502d9-143b-450e-9096-fc733d804245","Type":"ContainerDied","Data":"086949254fc3006849ed29056101c77418b572cfaea251f69a9ca92b37ac9287"} Oct 30 14:22:19.293303 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.293153 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="086949254fc3006849ed29056101c77418b572cfaea251f69a9ca92b37ac9287" Oct 30 14:22:19.293303 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.293103 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5" Oct 30 14:22:19.885009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.884978 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g"] Oct 30 14:22:19.885497 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.885484 4968 cpu_manager.go:414] "containerMap: removing container" podUID="e5a502d9-143b-450e-9096-fc733d804245" containerName="revoke-keys" Oct 30 14:22:19.885544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.885498 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a502d9-143b-450e-9096-fc733d804245" containerName="revoke-keys" Oct 30 14:22:19.885595 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.885585 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a502d9-143b-450e-9096-fc733d804245" containerName="revoke-keys" Oct 30 14:22:19.889212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.889181 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:19.918952 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.918922 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:22:19.919158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.919135 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:22:19.919251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.919052 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:22:19.919419 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.919399 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:22:19.923003 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.922979 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g"] Oct 30 14:22:19.989918 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.989879 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47155609-89n7g\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:19.990116 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.989958 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47155609-89n7g\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:19.990159 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:19.990115 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxtkr\" (UniqueName: \"kubernetes.io/projected/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-kube-api-access-gxtkr\") pod \"maas-api-revoke-sub-47155609-89n7g\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:20.091633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:20.091592 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47155609-89n7g\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:20.091845 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:20.091672 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47155609-89n7g\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:20.091845 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:20.091817 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxtkr\" (UniqueName: \"kubernetes.io/projected/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-kube-api-access-gxtkr\") pod \"maas-api-revoke-sub-47155609-89n7g\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:20.092435 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:20.092411 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47155609-89n7g\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:20.094041 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:20.094019 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47155609-89n7g\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:20.114268 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:20.114240 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxtkr\" (UniqueName: \"kubernetes.io/projected/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-kube-api-access-gxtkr\") pod \"maas-api-revoke-sub-47155609-89n7g\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:20.205287 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:20.205201 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:20.445169 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:22:20.445132 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:22:20.511245 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:20.511216 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g"] Oct 30 14:22:20.513310 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:20.513278 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6edd68_f51f_4e9c_b814_2af7b22c3b53.slice/crio-fec6fce34436f56bf5558e643d1236050b012b0dc8f56838c596133e3aa3eddb WatchSource:0}: Error finding container fec6fce34436f56bf5558e643d1236050b012b0dc8f56838c596133e3aa3eddb: Status 404 returned error can't find the container with id fec6fce34436f56bf5558e643d1236050b012b0dc8f56838c596133e3aa3eddb Oct 30 14:22:21.065792 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.063262 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc"] Oct 30 14:22:21.076439 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.076107 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:21.087012 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.086366 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc"] Oct 30 14:22:21.105286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.105235 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwq4t\" (UniqueName: \"kubernetes.io/projected/8f34edf8-b354-4392-ac1e-4a65dde459b0-kube-api-access-xwq4t\") pod \"maas-api-revoke-sub-6009ef72-dtprc\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:21.105431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.105313 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6009ef72-dtprc\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:21.105431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.105382 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6009ef72-dtprc\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:21.206740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.206679 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwq4t\" (UniqueName: \"kubernetes.io/projected/8f34edf8-b354-4392-ac1e-4a65dde459b0-kube-api-access-xwq4t\") pod \"maas-api-revoke-sub-6009ef72-dtprc\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:21.206929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.206751 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6009ef72-dtprc\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:21.206929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.206812 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6009ef72-dtprc\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:21.207440 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.207412 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6009ef72-dtprc\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:21.209328 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.209308 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6009ef72-dtprc\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:21.228349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.228316 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwq4t\" (UniqueName: \"kubernetes.io/projected/8f34edf8-b354-4392-ac1e-4a65dde459b0-kube-api-access-xwq4t\") pod \"maas-api-revoke-sub-6009ef72-dtprc\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:21.312193 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.312160 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" event={"ID":"ea6edd68-f51f-4e9c-b814-2af7b22c3b53","Type":"ContainerStarted","Data":"ead147a7aa59d4833cf6976320ce3a4e008c1176cf9ccad4071b9c75429d8db8"} Oct 30 14:22:21.312347 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.312211 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" event={"ID":"ea6edd68-f51f-4e9c-b814-2af7b22c3b53","Type":"ContainerStarted","Data":"fec6fce34436f56bf5558e643d1236050b012b0dc8f56838c596133e3aa3eddb"} Oct 30 14:22:21.351792 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.351742 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" podStartSLOduration=2.351727664 podStartE2EDuration="2.351727664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:21.348757964 +0000 UTC m=+936.974201337" watchObservedRunningTime="2026-10-30 14:22:21.351727664 +0000 UTC m=+936.977171031" Oct 30 14:22:21.395659 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.395630 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:21.699522 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:21.699442 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc"] Oct 30 14:22:21.703556 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:21.703524 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f34edf8_b354_4392_ac1e_4a65dde459b0.slice/crio-0f24eec12354e51e49d2015dd7b1ea57a307b256ac185ba7d001ee3d7f0f1bed WatchSource:0}: Error finding container 0f24eec12354e51e49d2015dd7b1ea57a307b256ac185ba7d001ee3d7f0f1bed: Status 404 returned error can't find the container with id 0f24eec12354e51e49d2015dd7b1ea57a307b256ac185ba7d001ee3d7f0f1bed Oct 30 14:22:22.319432 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:22.319348 4968 generic.go:356] "Generic (PLEG): container finished" podID="ea6edd68-f51f-4e9c-b814-2af7b22c3b53" containerID="ead147a7aa59d4833cf6976320ce3a4e008c1176cf9ccad4071b9c75429d8db8" exitCode=0 Oct 30 14:22:22.319838 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:22.319432 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" event={"ID":"ea6edd68-f51f-4e9c-b814-2af7b22c3b53","Type":"ContainerDied","Data":"ead147a7aa59d4833cf6976320ce3a4e008c1176cf9ccad4071b9c75429d8db8"} Oct 30 14:22:22.321026 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:22.321002 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" event={"ID":"8f34edf8-b354-4392-ac1e-4a65dde459b0","Type":"ContainerStarted","Data":"1423985132e0f0a7b39a26a99ff513c13842a5765a84be1527efe7fd2681d29a"} Oct 30 14:22:22.321138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:22.321040 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" event={"ID":"8f34edf8-b354-4392-ac1e-4a65dde459b0","Type":"ContainerStarted","Data":"0f24eec12354e51e49d2015dd7b1ea57a307b256ac185ba7d001ee3d7f0f1bed"} Oct 30 14:22:22.395739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:22.395203 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" podStartSLOduration=1.395183822 podStartE2EDuration="1.395183822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:22.393253822 +0000 UTC m=+938.018697195" watchObservedRunningTime="2026-10-30 14:22:22.395183822 +0000 UTC m=+938.020627195" Oct 30 14:22:23.328794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.328761 4968 generic.go:356] "Generic (PLEG): container finished" podID="8f34edf8-b354-4392-ac1e-4a65dde459b0" containerID="1423985132e0f0a7b39a26a99ff513c13842a5765a84be1527efe7fd2681d29a" exitCode=0 Oct 30 14:22:23.329215 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.328890 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" event={"ID":"8f34edf8-b354-4392-ac1e-4a65dde459b0","Type":"ContainerDied","Data":"1423985132e0f0a7b39a26a99ff513c13842a5765a84be1527efe7fd2681d29a"} Oct 30 14:22:23.489423 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.489393 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:23.531046 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.531021 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-kube-api-access-gxtkr\" (UniqueName: \"kubernetes.io/projected/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-kube-api-access-gxtkr\") pod \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " Oct 30 14:22:23.531211 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.531104 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-xks-ca\") pod \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " Oct 30 14:22:23.531211 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.531191 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-openshift-ca\") pod \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\" (UID: \"ea6edd68-f51f-4e9c-b814-2af7b22c3b53\") " Oct 30 14:22:23.531608 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.531575 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-openshift-ca" pod "ea6edd68-f51f-4e9c-b814-2af7b22c3b53" (UID: "ea6edd68-f51f-4e9c-b814-2af7b22c3b53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:23.531739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.531682 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:23.533848 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.533815 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-kube-api-access-gxtkr" pod "ea6edd68-f51f-4e9c-b814-2af7b22c3b53" (UID: "ea6edd68-f51f-4e9c-b814-2af7b22c3b53"). InnerVolumeSpecName "kube-api-access-gxtkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:23.533962 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.533931 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-xks-ca" pod "ea6edd68-f51f-4e9c-b814-2af7b22c3b53" (UID: "ea6edd68-f51f-4e9c-b814-2af7b22c3b53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:23.633600 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.633556 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxtkr\" (UniqueName: \"kubernetes.io/projected/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-kube-api-access-gxtkr\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:23.633600 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:23.633593 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6edd68-f51f-4e9c-b814-2af7b22c3b53-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:24.335987 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.335956 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" Oct 30 14:22:24.336479 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.335990 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g" event={"ID":"ea6edd68-f51f-4e9c-b814-2af7b22c3b53","Type":"ContainerDied","Data":"fec6fce34436f56bf5558e643d1236050b012b0dc8f56838c596133e3aa3eddb"} Oct 30 14:22:24.336479 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.336044 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fec6fce34436f56bf5558e643d1236050b012b0dc8f56838c596133e3aa3eddb" Oct 30 14:22:24.504013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.503983 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:24.521478 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.521449 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl"] Oct 30 14:22:24.522440 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.522419 4968 cpu_manager.go:414] "containerMap: removing container" podUID="8f34edf8-b354-4392-ac1e-4a65dde459b0" containerName="revoke-keys" Oct 30 14:22:24.522440 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.522439 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f34edf8-b354-4392-ac1e-4a65dde459b0" containerName="revoke-keys" Oct 30 14:22:24.522594 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.522454 4968 cpu_manager.go:414] "containerMap: removing container" podUID="ea6edd68-f51f-4e9c-b814-2af7b22c3b53" containerName="revoke-keys" Oct 30 14:22:24.522594 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.522461 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6edd68-f51f-4e9c-b814-2af7b22c3b53" containerName="revoke-keys" Oct 30 14:22:24.522594 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.522581 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f34edf8-b354-4392-ac1e-4a65dde459b0" containerName="revoke-keys" Oct 30 14:22:24.522776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.522595 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6edd68-f51f-4e9c-b814-2af7b22c3b53" containerName="revoke-keys" Oct 30 14:22:24.529676 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.527464 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:24.545145 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.545117 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-openshift-ca\") pod \"8f34edf8-b354-4392-ac1e-4a65dde459b0\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " Oct 30 14:22:24.545375 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.545204 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f34edf8-b354-4392-ac1e-4a65dde459b0-kube-api-access-xwq4t\" (UniqueName: \"kubernetes.io/projected/8f34edf8-b354-4392-ac1e-4a65dde459b0-kube-api-access-xwq4t\") pod \"8f34edf8-b354-4392-ac1e-4a65dde459b0\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " Oct 30 14:22:24.545375 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.545229 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-xks-ca\") pod \"8f34edf8-b354-4392-ac1e-4a65dde459b0\" (UID: \"8f34edf8-b354-4392-ac1e-4a65dde459b0\") " Oct 30 14:22:24.546100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.546074 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-openshift-ca" pod "8f34edf8-b354-4392-ac1e-4a65dde459b0" (UID: "8f34edf8-b354-4392-ac1e-4a65dde459b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:24.547856 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.547432 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl"] Oct 30 14:22:24.551025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.550997 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-xks-ca" pod "8f34edf8-b354-4392-ac1e-4a65dde459b0" (UID: "8f34edf8-b354-4392-ac1e-4a65dde459b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:24.551272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.551238 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f34edf8-b354-4392-ac1e-4a65dde459b0-kube-api-access-xwq4t" pod "8f34edf8-b354-4392-ac1e-4a65dde459b0" (UID: "8f34edf8-b354-4392-ac1e-4a65dde459b0"). InnerVolumeSpecName "kube-api-access-xwq4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:24.663797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.663432 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjqf\" (UniqueName: \"kubernetes.io/projected/ae8b52a4-e6ca-4416-be2e-629351fc37f9-kube-api-access-5bjqf\") pod \"maas-api-revoke-sub-20be9e14-qftcl\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:24.663797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.663508 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20be9e14-qftcl\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:24.663797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.663532 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20be9e14-qftcl\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:24.663797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.663590 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:24.663797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.663606 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwq4t\" (UniqueName: \"kubernetes.io/projected/8f34edf8-b354-4392-ac1e-4a65dde459b0-kube-api-access-xwq4t\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:24.663797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.663620 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f34edf8-b354-4392-ac1e-4a65dde459b0-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:24.764182 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.764148 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjqf\" (UniqueName: \"kubernetes.io/projected/ae8b52a4-e6ca-4416-be2e-629351fc37f9-kube-api-access-5bjqf\") pod \"maas-api-revoke-sub-20be9e14-qftcl\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:24.764371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.764200 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20be9e14-qftcl\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:24.764371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.764223 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20be9e14-qftcl\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:24.765003 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.764977 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20be9e14-qftcl\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:24.766929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.766909 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20be9e14-qftcl\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:24.785961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.785936 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjqf\" (UniqueName: \"kubernetes.io/projected/ae8b52a4-e6ca-4416-be2e-629351fc37f9-kube-api-access-5bjqf\") pod \"maas-api-revoke-sub-20be9e14-qftcl\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:24.844064 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:24.844038 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:25.157811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:25.157742 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl"] Oct 30 14:22:25.160583 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:25.160555 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8b52a4_e6ca_4416_be2e_629351fc37f9.slice/crio-b03cb218940d44356e196b9e35f10e1fe9cdd333af4f57b176b716d771dbf76f WatchSource:0}: Error finding container b03cb218940d44356e196b9e35f10e1fe9cdd333af4f57b176b716d771dbf76f: Status 404 returned error can't find the container with id b03cb218940d44356e196b9e35f10e1fe9cdd333af4f57b176b716d771dbf76f Oct 30 14:22:25.344773 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:25.344733 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" event={"ID":"ae8b52a4-e6ca-4416-be2e-629351fc37f9","Type":"ContainerStarted","Data":"b03cb218940d44356e196b9e35f10e1fe9cdd333af4f57b176b716d771dbf76f"} Oct 30 14:22:25.346315 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:25.346288 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" Oct 30 14:22:25.346440 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:25.346293 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc" event={"ID":"8f34edf8-b354-4392-ac1e-4a65dde459b0","Type":"ContainerDied","Data":"0f24eec12354e51e49d2015dd7b1ea57a307b256ac185ba7d001ee3d7f0f1bed"} Oct 30 14:22:25.346440 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:25.346423 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f24eec12354e51e49d2015dd7b1ea57a307b256ac185ba7d001ee3d7f0f1bed" Oct 30 14:22:26.378765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.358152 4968 generic.go:356] "Generic (PLEG): container finished" podID="ae8b52a4-e6ca-4416-be2e-629351fc37f9" containerID="9a5033521a1c1cab8af0b57811ff9070b47991ebcf82670bc990e48324ade7b7" exitCode=0 Oct 30 14:22:26.378765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.358273 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" event={"ID":"ae8b52a4-e6ca-4416-be2e-629351fc37f9","Type":"ContainerDied","Data":"9a5033521a1c1cab8af0b57811ff9070b47991ebcf82670bc990e48324ade7b7"} Oct 30 14:22:26.450481 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.450446 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57"] Oct 30 14:22:26.455160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.455135 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:26.478241 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.478213 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57"] Oct 30 14:22:26.584465 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.584426 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-138eee7f-9rf57\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:26.584644 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.584626 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fjlb\" (UniqueName: \"kubernetes.io/projected/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-kube-api-access-5fjlb\") pod \"maas-api-revoke-sub-138eee7f-9rf57\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:26.584698 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.584675 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-138eee7f-9rf57\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:26.685641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.685558 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-138eee7f-9rf57\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:26.685804 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.685669 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fjlb\" (UniqueName: \"kubernetes.io/projected/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-kube-api-access-5fjlb\") pod \"maas-api-revoke-sub-138eee7f-9rf57\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:26.685804 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.685698 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-138eee7f-9rf57\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:26.686409 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.686379 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-138eee7f-9rf57\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:26.688162 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.688132 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-138eee7f-9rf57\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:26.707549 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.707520 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fjlb\" (UniqueName: \"kubernetes.io/projected/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-kube-api-access-5fjlb\") pod \"maas-api-revoke-sub-138eee7f-9rf57\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:26.768443 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:26.768412 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:27.089506 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.089471 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57"] Oct 30 14:22:27.095539 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:27.095506 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67cc2eb0_e6ac_412e_8618_a8ae5489c6a4.slice/crio-3b06210fa14d7dd49b41b8a10e8748babfdd850364327a8b75a6a8a77f1ce9b7 WatchSource:0}: Error finding container 3b06210fa14d7dd49b41b8a10e8748babfdd850364327a8b75a6a8a77f1ce9b7: Status 404 returned error can't find the container with id 3b06210fa14d7dd49b41b8a10e8748babfdd850364327a8b75a6a8a77f1ce9b7 Oct 30 14:22:27.366019 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.365936 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" event={"ID":"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4","Type":"ContainerStarted","Data":"3b06210fa14d7dd49b41b8a10e8748babfdd850364327a8b75a6a8a77f1ce9b7"} Oct 30 14:22:27.492397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.492368 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:27.603666 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.603631 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-openshift-ca\") pod \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " Oct 30 14:22:27.603846 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.603698 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-xks-ca\") pod \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " Oct 30 14:22:27.603908 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.603864 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8b52a4-e6ca-4416-be2e-629351fc37f9-kube-api-access-5bjqf\" (UniqueName: \"kubernetes.io/projected/ae8b52a4-e6ca-4416-be2e-629351fc37f9-kube-api-access-5bjqf\") pod \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\" (UID: \"ae8b52a4-e6ca-4416-be2e-629351fc37f9\") " Oct 30 14:22:27.604117 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.604091 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-openshift-ca" pod "ae8b52a4-e6ca-4416-be2e-629351fc37f9" (UID: "ae8b52a4-e6ca-4416-be2e-629351fc37f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:27.604426 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.604395 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:27.606021 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.605991 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-xks-ca" pod "ae8b52a4-e6ca-4416-be2e-629351fc37f9" (UID: "ae8b52a4-e6ca-4416-be2e-629351fc37f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:27.606137 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.606039 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8b52a4-e6ca-4416-be2e-629351fc37f9-kube-api-access-5bjqf" pod "ae8b52a4-e6ca-4416-be2e-629351fc37f9" (UID: "ae8b52a4-e6ca-4416-be2e-629351fc37f9"). InnerVolumeSpecName "kube-api-access-5bjqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:27.705887 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.705853 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8b52a4-e6ca-4416-be2e-629351fc37f9-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:27.705887 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:27.705888 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bjqf\" (UniqueName: \"kubernetes.io/projected/ae8b52a4-e6ca-4416-be2e-629351fc37f9-kube-api-access-5bjqf\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:28.374219 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:28.374187 4968 generic.go:356] "Generic (PLEG): container finished" podID="67cc2eb0-e6ac-412e-8618-a8ae5489c6a4" containerID="411074a10307049b4821e1346eef2fc1c0de4f4ef1bcf91111abeeeb49f8988c" exitCode=0 Oct 30 14:22:28.374399 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:28.374270 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" event={"ID":"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4","Type":"ContainerDied","Data":"411074a10307049b4821e1346eef2fc1c0de4f4ef1bcf91111abeeeb49f8988c"} Oct 30 14:22:28.375656 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:28.375637 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" Oct 30 14:22:28.375778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:28.375636 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl" event={"ID":"ae8b52a4-e6ca-4416-be2e-629351fc37f9","Type":"ContainerDied","Data":"b03cb218940d44356e196b9e35f10e1fe9cdd333af4f57b176b716d771dbf76f"} Oct 30 14:22:28.375778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:28.375760 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b03cb218940d44356e196b9e35f10e1fe9cdd333af4f57b176b716d771dbf76f" Oct 30 14:22:29.157378 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:29.157343 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:22:29.543894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:29.543866 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:29.628866 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:29.628837 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-xks-ca\") pod \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " Oct 30 14:22:29.629022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:29.628987 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-openshift-ca\") pod \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " Oct 30 14:22:29.629106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:29.629024 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-kube-api-access-5fjlb\" (UniqueName: \"kubernetes.io/projected/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-kube-api-access-5fjlb\") pod \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\" (UID: \"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4\") " Oct 30 14:22:29.629427 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:29.629385 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-openshift-ca" pod "67cc2eb0-e6ac-412e-8618-a8ae5489c6a4" (UID: "67cc2eb0-e6ac-412e-8618-a8ae5489c6a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:29.631186 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:29.631159 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-xks-ca" pod "67cc2eb0-e6ac-412e-8618-a8ae5489c6a4" (UID: "67cc2eb0-e6ac-412e-8618-a8ae5489c6a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:29.631474 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:29.631456 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-kube-api-access-5fjlb" pod "67cc2eb0-e6ac-412e-8618-a8ae5489c6a4" (UID: "67cc2eb0-e6ac-412e-8618-a8ae5489c6a4"). InnerVolumeSpecName "kube-api-access-5fjlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:29.731060 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:29.730978 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:29.731060 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:29.731008 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fjlb\" (UniqueName: \"kubernetes.io/projected/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-kube-api-access-5fjlb\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:29.731060 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:29.731021 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:30.399741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:30.394840 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" Oct 30 14:22:30.399741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:30.394882 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57" event={"ID":"67cc2eb0-e6ac-412e-8618-a8ae5489c6a4","Type":"ContainerDied","Data":"3b06210fa14d7dd49b41b8a10e8748babfdd850364327a8b75a6a8a77f1ce9b7"} Oct 30 14:22:30.399741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:30.394944 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b06210fa14d7dd49b41b8a10e8748babfdd850364327a8b75a6a8a77f1ce9b7" Oct 30 14:22:31.409177 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.409137 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws"] Oct 30 14:22:31.410381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.410352 4968 cpu_manager.go:414] "containerMap: removing container" podUID="67cc2eb0-e6ac-412e-8618-a8ae5489c6a4" containerName="revoke-keys" Oct 30 14:22:31.410381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.410379 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67cc2eb0-e6ac-412e-8618-a8ae5489c6a4" containerName="revoke-keys" Oct 30 14:22:31.410564 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.410422 4968 cpu_manager.go:414] "containerMap: removing container" podUID="ae8b52a4-e6ca-4416-be2e-629351fc37f9" containerName="revoke-keys" Oct 30 14:22:31.410564 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.410431 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8b52a4-e6ca-4416-be2e-629351fc37f9" containerName="revoke-keys" Oct 30 14:22:31.410762 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.410745 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="67cc2eb0-e6ac-412e-8618-a8ae5489c6a4" containerName="revoke-keys" Oct 30 14:22:31.410823 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.410777 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8b52a4-e6ca-4416-be2e-629351fc37f9" containerName="revoke-keys" Oct 30 14:22:31.419311 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.419286 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.438384 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.438346 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws"] Oct 30 14:22:31.470663 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.470628 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-717995-openshift-default-dockercfg-h5tlx\"" Oct 30 14:22:31.556454 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.556421 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-certs\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.556627 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.556463 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-podinfo\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.556627 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.556484 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a2483e-29a2-4369-8d98-3e0860468bb0-istiod-ca-cert\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.556627 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.556545 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-data\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.556627 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.556586 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-credential-socket\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.556781 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.556643 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-token\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.556781 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.556691 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-envoy\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.556781 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.556769 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xq5\" (UniqueName: \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-kube-api-access-f4xq5\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.556879 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.556800 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-socket\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.657834 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.657801 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-socket\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.658188 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.658160 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-certs\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.658401 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.658207 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-socket\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.658401 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.658243 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-podinfo\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.658401 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.658273 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a2483e-29a2-4369-8d98-3e0860468bb0-istiod-ca-cert\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.658401 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.658306 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-data\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.658615 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.658416 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-credential-socket\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.658615 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.658436 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-certs\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.658615 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.658447 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-token\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.658615 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.658521 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-envoy\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.658615 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.658578 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xq5\" (UniqueName: \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-kube-api-access-f4xq5\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.658889 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.658643 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-data\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.659103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.659075 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a2483e-29a2-4369-8d98-3e0860468bb0-istiod-ca-cert\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.659239 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.659084 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-credential-socket\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.661012 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.660990 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-podinfo\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.661105 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.660990 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-envoy\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.683641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.683613 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-token\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.683899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.683875 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xq5\" (UniqueName: \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-kube-api-access-f4xq5\") pod \"e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:31.746866 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:31.746828 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:32.071067 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:32.070940 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws"] Oct 30 14:22:32.073589 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:32.073561 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a2483e_29a2_4369_8d98_3e0860468bb0.slice/crio-96f9efeda017c0224da8755b934064e92bc0888f9f82d0488b9274b486d7deeb WatchSource:0}: Error finding container 96f9efeda017c0224da8755b934064e92bc0888f9f82d0488b9274b486d7deeb: Status 404 returned error can't find the container with id 96f9efeda017c0224da8755b934064e92bc0888f9f82d0488b9274b486d7deeb Oct 30 14:22:32.076029 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:32.075997 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:22:32.076111 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:32.076065 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:22:32.076111 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:32.076095 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:22:32.431170 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:32.431134 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" event={"ID":"92a2483e-29a2-4369-8d98-3e0860468bb0","Type":"ContainerStarted","Data":"b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2"} Oct 30 14:22:32.431659 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:32.431184 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" event={"ID":"92a2483e-29a2-4369-8d98-3e0860468bb0","Type":"ContainerStarted","Data":"96f9efeda017c0224da8755b934064e92bc0888f9f82d0488b9274b486d7deeb"} Oct 30 14:22:32.475834 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:32.475782 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" podStartSLOduration=1.4757671079999999 podStartE2EDuration="1.475767108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:32.472619211 +0000 UTC m=+948.098062584" watchObservedRunningTime="2026-10-30 14:22:32.475767108 +0000 UTC m=+948.101210535" Oct 30 14:22:32.747301 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:32.747231 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:33.749204 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:33.749160 4968 patch_prober.go:29] interesting pod/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:22:33.749687 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:33.749227 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" podUID="92a2483e-29a2-4369-8d98-3e0860468bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 14:22:34.308042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.308001 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg"] Oct 30 14:22:34.320824 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.320793 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg"] Oct 30 14:22:34.320977 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.320893 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:34.343864 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.343836 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:22:34.344010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.343916 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:22:34.344010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.343926 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:22:34.344100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.344011 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:22:34.413214 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.413180 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68e9f918-kb2qg\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:34.413392 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.413364 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68e9f918-kb2qg\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:34.413472 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.413456 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5tx8\" (UniqueName: \"kubernetes.io/projected/56cb76e2-0b34-4d52-9297-abab9c9c53ef-kube-api-access-f5tx8\") pod \"maas-api-revoke-sub-68e9f918-kb2qg\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:34.514115 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.514077 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68e9f918-kb2qg\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:34.514308 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.514228 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5tx8\" (UniqueName: \"kubernetes.io/projected/56cb76e2-0b34-4d52-9297-abab9c9c53ef-kube-api-access-f5tx8\") pod \"maas-api-revoke-sub-68e9f918-kb2qg\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:34.514379 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.514341 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68e9f918-kb2qg\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:34.515064 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.514697 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68e9f918-kb2qg\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:34.519611 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.518417 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68e9f918-kb2qg\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:34.537004 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.536974 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5tx8\" (UniqueName: \"kubernetes.io/projected/56cb76e2-0b34-4d52-9297-abab9c9c53ef-kube-api-access-f5tx8\") pod \"maas-api-revoke-sub-68e9f918-kb2qg\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:34.638938 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.638908 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:34.750206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.750104 4968 patch_prober.go:29] interesting pod/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:22:34.750206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.750154 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" podUID="92a2483e-29a2-4369-8d98-3e0860468bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 14:22:34.753369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.753189 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz"] Oct 30 14:22:34.759285 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.759261 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:34.769741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.767309 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz"] Oct 30 14:22:34.924330 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.924250 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91920ce-dgnkz\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:34.924526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.924378 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91920ce-dgnkz\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:34.924526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.924460 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bb64\" (UniqueName: \"kubernetes.io/projected/b0373cdd-31d6-464c-8baf-c8f422f6a00b-kube-api-access-2bb64\") pod \"maas-api-revoke-sub-b91920ce-dgnkz\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:34.970045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:34.970021 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg"] Oct 30 14:22:34.972915 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:34.972889 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cb76e2_0b34_4d52_9297_abab9c9c53ef.slice/crio-db38f133c5f7b69ec9823f9305e34958a636b7b733d82e8e2b1b395c08cd9d25 WatchSource:0}: Error finding container db38f133c5f7b69ec9823f9305e34958a636b7b733d82e8e2b1b395c08cd9d25: Status 404 returned error can't find the container with id db38f133c5f7b69ec9823f9305e34958a636b7b733d82e8e2b1b395c08cd9d25 Oct 30 14:22:35.026248 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.026221 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91920ce-dgnkz\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:35.026356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.026281 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91920ce-dgnkz\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:35.026356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.026343 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bb64\" (UniqueName: \"kubernetes.io/projected/b0373cdd-31d6-464c-8baf-c8f422f6a00b-kube-api-access-2bb64\") pod \"maas-api-revoke-sub-b91920ce-dgnkz\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:35.026847 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.026826 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91920ce-dgnkz\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:35.028450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.028433 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91920ce-dgnkz\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:35.047624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.047600 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bb64\" (UniqueName: \"kubernetes.io/projected/b0373cdd-31d6-464c-8baf-c8f422f6a00b-kube-api-access-2bb64\") pod \"maas-api-revoke-sub-b91920ce-dgnkz\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:35.114300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.114260 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:35.452153 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.452068 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" event={"ID":"56cb76e2-0b34-4d52-9297-abab9c9c53ef","Type":"ContainerStarted","Data":"0f497c55a555b893d6bf6033306edc65ac3c780213e89295958e942a16252da4"} Oct 30 14:22:35.452153 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.452132 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" event={"ID":"56cb76e2-0b34-4d52-9297-abab9c9c53ef","Type":"ContainerStarted","Data":"db38f133c5f7b69ec9823f9305e34958a636b7b733d82e8e2b1b395c08cd9d25"} Oct 30 14:22:35.506746 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.506453 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" podStartSLOduration=1.506430758 podStartE2EDuration="1.506430758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:35.494158577 +0000 UTC m=+951.119601954" watchObservedRunningTime="2026-10-30 14:22:35.506430758 +0000 UTC m=+951.131874133" Oct 30 14:22:35.565450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.565426 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz"] Oct 30 14:22:35.568490 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:35.568458 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0373cdd_31d6_464c_8baf_c8f422f6a00b.slice/crio-b8160efd5686e07aeccc2378bdfebb69b533f3150f00e06227055a43469f4c6a WatchSource:0}: Error finding container b8160efd5686e07aeccc2378bdfebb69b533f3150f00e06227055a43469f4c6a: Status 404 returned error can't find the container with id b8160efd5686e07aeccc2378bdfebb69b533f3150f00e06227055a43469f4c6a Oct 30 14:22:35.747742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.747631 4968 patch_prober.go:29] interesting pod/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:22:35.747742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:35.747682 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" podUID="92a2483e-29a2-4369-8d98-3e0860468bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 14:22:36.460335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:36.460255 4968 generic.go:356] "Generic (PLEG): container finished" podID="56cb76e2-0b34-4d52-9297-abab9c9c53ef" containerID="0f497c55a555b893d6bf6033306edc65ac3c780213e89295958e942a16252da4" exitCode=0 Oct 30 14:22:36.460710 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:36.460330 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" event={"ID":"56cb76e2-0b34-4d52-9297-abab9c9c53ef","Type":"ContainerDied","Data":"0f497c55a555b893d6bf6033306edc65ac3c780213e89295958e942a16252da4"} Oct 30 14:22:36.461972 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:36.461947 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" event={"ID":"b0373cdd-31d6-464c-8baf-c8f422f6a00b","Type":"ContainerStarted","Data":"87839ac780134bd9cf8040e9e7ffd547f8a3739c40d39e3025ac02faf466e123"} Oct 30 14:22:36.462085 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:36.461989 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" event={"ID":"b0373cdd-31d6-464c-8baf-c8f422f6a00b","Type":"ContainerStarted","Data":"b8160efd5686e07aeccc2378bdfebb69b533f3150f00e06227055a43469f4c6a"} Oct 30 14:22:36.537424 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:36.537377 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" podStartSLOduration=2.53736021 podStartE2EDuration="2.53736021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:36.534286702 +0000 UTC m=+952.159730076" watchObservedRunningTime="2026-10-30 14:22:36.53736021 +0000 UTC m=+952.162803582" Oct 30 14:22:36.747916 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:36.747879 4968 patch_prober.go:29] interesting pod/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:22:36.748087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:36.747930 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" podUID="92a2483e-29a2-4369-8d98-3e0860468bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 14:22:37.472375 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.472342 4968 generic.go:356] "Generic (PLEG): container finished" podID="b0373cdd-31d6-464c-8baf-c8f422f6a00b" containerID="87839ac780134bd9cf8040e9e7ffd547f8a3739c40d39e3025ac02faf466e123" exitCode=0 Oct 30 14:22:37.472811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.472415 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" event={"ID":"b0373cdd-31d6-464c-8baf-c8f422f6a00b","Type":"ContainerDied","Data":"87839ac780134bd9cf8040e9e7ffd547f8a3739c40d39e3025ac02faf466e123"} Oct 30 14:22:37.708248 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.708223 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:37.749824 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.748006 4968 patch_prober.go:29] interesting pod/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:22:37.749824 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.748048 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" podUID="92a2483e-29a2-4369-8d98-3e0860468bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 14:22:37.865100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.865069 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-xks-ca\") pod \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " Oct 30 14:22:37.865260 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.865157 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-openshift-ca\") pod \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " Oct 30 14:22:37.865260 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.865188 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56cb76e2-0b34-4d52-9297-abab9c9c53ef-kube-api-access-f5tx8\" (UniqueName: \"kubernetes.io/projected/56cb76e2-0b34-4d52-9297-abab9c9c53ef-kube-api-access-f5tx8\") pod \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\" (UID: \"56cb76e2-0b34-4d52-9297-abab9c9c53ef\") " Oct 30 14:22:37.865522 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.865494 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-openshift-ca" pod "56cb76e2-0b34-4d52-9297-abab9c9c53ef" (UID: "56cb76e2-0b34-4d52-9297-abab9c9c53ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:37.867256 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.867230 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-xks-ca" pod "56cb76e2-0b34-4d52-9297-abab9c9c53ef" (UID: "56cb76e2-0b34-4d52-9297-abab9c9c53ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:37.867361 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.867333 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56cb76e2-0b34-4d52-9297-abab9c9c53ef-kube-api-access-f5tx8" pod "56cb76e2-0b34-4d52-9297-abab9c9c53ef" (UID: "56cb76e2-0b34-4d52-9297-abab9c9c53ef"). InnerVolumeSpecName "kube-api-access-f5tx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:37.966459 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.966423 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:37.966459 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.966452 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56cb76e2-0b34-4d52-9297-abab9c9c53ef-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:37.966652 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:37.966467 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5tx8\" (UniqueName: \"kubernetes.io/projected/56cb76e2-0b34-4d52-9297-abab9c9c53ef-kube-api-access-f5tx8\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:38.356307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.356273 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n"] Oct 30 14:22:38.357131 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.357112 4968 cpu_manager.go:414] "containerMap: removing container" podUID="56cb76e2-0b34-4d52-9297-abab9c9c53ef" containerName="revoke-keys" Oct 30 14:22:38.357190 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.357135 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56cb76e2-0b34-4d52-9297-abab9c9c53ef" containerName="revoke-keys" Oct 30 14:22:38.357315 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.357302 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="56cb76e2-0b34-4d52-9297-abab9c9c53ef" containerName="revoke-keys" Oct 30 14:22:38.361288 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.361268 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:38.381755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.381299 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n"] Oct 30 14:22:38.471941 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.471910 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77051e7e-r4q5n\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:38.471941 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.471945 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77051e7e-r4q5n\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:38.472174 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.472095 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq46p\" (UniqueName: \"kubernetes.io/projected/189c32fe-20bf-45e5-9aba-154f23d5e895-kube-api-access-tq46p\") pod \"maas-api-revoke-sub-77051e7e-r4q5n\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:38.482623 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.482598 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" Oct 30 14:22:38.483093 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.482625 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg" event={"ID":"56cb76e2-0b34-4d52-9297-abab9c9c53ef","Type":"ContainerDied","Data":"db38f133c5f7b69ec9823f9305e34958a636b7b733d82e8e2b1b395c08cd9d25"} Oct 30 14:22:38.483093 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.482686 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db38f133c5f7b69ec9823f9305e34958a636b7b733d82e8e2b1b395c08cd9d25" Oct 30 14:22:38.574483 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.574446 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq46p\" (UniqueName: \"kubernetes.io/projected/189c32fe-20bf-45e5-9aba-154f23d5e895-kube-api-access-tq46p\") pod \"maas-api-revoke-sub-77051e7e-r4q5n\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:38.574653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.574585 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77051e7e-r4q5n\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:38.574653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.574615 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77051e7e-r4q5n\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:38.575422 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.575395 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77051e7e-r4q5n\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:38.579021 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.578999 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77051e7e-r4q5n\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:38.610446 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.610352 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq46p\" (UniqueName: \"kubernetes.io/projected/189c32fe-20bf-45e5-9aba-154f23d5e895-kube-api-access-tq46p\") pod \"maas-api-revoke-sub-77051e7e-r4q5n\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:38.645807 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.645783 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:38.674892 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.674865 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:38.782733 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.782219 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-openshift-ca\") pod \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " Oct 30 14:22:38.782733 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.782269 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0373cdd-31d6-464c-8baf-c8f422f6a00b-kube-api-access-2bb64\" (UniqueName: \"kubernetes.io/projected/b0373cdd-31d6-464c-8baf-c8f422f6a00b-kube-api-access-2bb64\") pod \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " Oct 30 14:22:38.782733 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.782341 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-xks-ca\") pod \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\" (UID: \"b0373cdd-31d6-464c-8baf-c8f422f6a00b\") " Oct 30 14:22:38.784059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.783483 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-openshift-ca" pod "b0373cdd-31d6-464c-8baf-c8f422f6a00b" (UID: "b0373cdd-31d6-464c-8baf-c8f422f6a00b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:38.786534 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.785242 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-xks-ca" pod "b0373cdd-31d6-464c-8baf-c8f422f6a00b" (UID: "b0373cdd-31d6-464c-8baf-c8f422f6a00b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:38.787418 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.787322 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0373cdd-31d6-464c-8baf-c8f422f6a00b-kube-api-access-2bb64" pod "b0373cdd-31d6-464c-8baf-c8f422f6a00b" (UID: "b0373cdd-31d6-464c-8baf-c8f422f6a00b"). InnerVolumeSpecName "kube-api-access-2bb64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:38.884568 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.884530 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:38.884707 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.884572 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bb64\" (UniqueName: \"kubernetes.io/projected/b0373cdd-31d6-464c-8baf-c8f422f6a00b-kube-api-access-2bb64\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:38.884707 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.884588 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0373cdd-31d6-464c-8baf-c8f422f6a00b-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:38.936854 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.936266 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:38.955748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.947194 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:38.955748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:38.951858 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:22:39.030606 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.030575 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n"] Oct 30 14:22:39.032455 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:39.032422 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189c32fe_20bf_45e5_9aba_154f23d5e895.slice/crio-4720b1c3c27193cc78f634f05d97f0a955ca3554a1f1bb3612c83f07e906e449 WatchSource:0}: Error finding container 4720b1c3c27193cc78f634f05d97f0a955ca3554a1f1bb3612c83f07e906e449: Status 404 returned error can't find the container with id 4720b1c3c27193cc78f634f05d97f0a955ca3554a1f1bb3612c83f07e906e449 Oct 30 14:22:39.112355 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.112327 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m"] Oct 30 14:22:39.112930 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.112912 4968 cpu_manager.go:414] "containerMap: removing container" podUID="b0373cdd-31d6-464c-8baf-c8f422f6a00b" containerName="revoke-keys" Oct 30 14:22:39.112930 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.112930 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0373cdd-31d6-464c-8baf-c8f422f6a00b" containerName="revoke-keys" Oct 30 14:22:39.113124 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.113111 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0373cdd-31d6-464c-8baf-c8f422f6a00b" containerName="revoke-keys" Oct 30 14:22:39.117802 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.117782 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:39.136212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.136151 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m"] Oct 30 14:22:39.192220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.192169 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd222ee-fw64m\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:39.192534 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.192507 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd222ee-fw64m\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:39.192874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.192663 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md26z\" (UniqueName: \"kubernetes.io/projected/308ea75f-c712-4435-a0b7-5134ff7bd65c-kube-api-access-md26z\") pod \"maas-api-revoke-sub-edd222ee-fw64m\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:39.293746 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.293683 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd222ee-fw64m\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:39.293894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.293761 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd222ee-fw64m\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:39.293894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.293798 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md26z\" (UniqueName: \"kubernetes.io/projected/308ea75f-c712-4435-a0b7-5134ff7bd65c-kube-api-access-md26z\") pod \"maas-api-revoke-sub-edd222ee-fw64m\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:39.294419 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.294394 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd222ee-fw64m\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:39.296070 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.296046 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd222ee-fw64m\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:39.317189 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.317165 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md26z\" (UniqueName: \"kubernetes.io/projected/308ea75f-c712-4435-a0b7-5134ff7bd65c-kube-api-access-md26z\") pod \"maas-api-revoke-sub-edd222ee-fw64m\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:39.432262 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.432239 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:39.491918 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.491882 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" event={"ID":"189c32fe-20bf-45e5-9aba-154f23d5e895","Type":"ContainerStarted","Data":"4720b1c3c27193cc78f634f05d97f0a955ca3554a1f1bb3612c83f07e906e449"} Oct 30 14:22:39.499131 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.497958 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" Oct 30 14:22:39.499525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.499442 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz" event={"ID":"b0373cdd-31d6-464c-8baf-c8f422f6a00b","Type":"ContainerDied","Data":"b8160efd5686e07aeccc2378bdfebb69b533f3150f00e06227055a43469f4c6a"} Oct 30 14:22:39.499525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.499502 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8160efd5686e07aeccc2378bdfebb69b533f3150f00e06227055a43469f4c6a" Oct 30 14:22:39.749668 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:39.749473 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m"] Oct 30 14:22:39.752088 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:39.752052 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308ea75f_c712_4435_a0b7_5134ff7bd65c.slice/crio-ad5b8d7155ef306cb399c3096d119cbf5d0092aea05d63af1ef9a6ffcc403747 WatchSource:0}: Error finding container ad5b8d7155ef306cb399c3096d119cbf5d0092aea05d63af1ef9a6ffcc403747: Status 404 returned error can't find the container with id ad5b8d7155ef306cb399c3096d119cbf5d0092aea05d63af1ef9a6ffcc403747 Oct 30 14:22:40.260420 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.260383 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb"] Oct 30 14:22:40.264302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.264283 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:40.286096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.286066 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb"] Oct 30 14:22:40.407007 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.406919 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c22439-xvkcb\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:40.407007 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.406956 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z28x\" (UniqueName: \"kubernetes.io/projected/fd3653da-8836-46e6-8c3b-dff86258aa8a-kube-api-access-2z28x\") pod \"maas-api-revoke-sub-d1c22439-xvkcb\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:40.407007 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.406998 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c22439-xvkcb\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:40.506367 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.506329 4968 generic.go:356] "Generic (PLEG): container finished" podID="189c32fe-20bf-45e5-9aba-154f23d5e895" containerID="adee60ba53b36842cdc4e1b361eb0285c7e0a2cc7981eaef63172c4ac3899995" exitCode=0 Oct 30 14:22:40.506861 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.506404 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" event={"ID":"189c32fe-20bf-45e5-9aba-154f23d5e895","Type":"ContainerDied","Data":"adee60ba53b36842cdc4e1b361eb0285c7e0a2cc7981eaef63172c4ac3899995"} Oct 30 14:22:40.507912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.507882 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c22439-xvkcb\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:40.508082 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.507920 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z28x\" (UniqueName: \"kubernetes.io/projected/fd3653da-8836-46e6-8c3b-dff86258aa8a-kube-api-access-2z28x\") pod \"maas-api-revoke-sub-d1c22439-xvkcb\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:40.508082 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.507985 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c22439-xvkcb\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:40.508331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.508233 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" event={"ID":"308ea75f-c712-4435-a0b7-5134ff7bd65c","Type":"ContainerStarted","Data":"168b650286d672ff74a373896576f917cc44e7d78f27262be6fe91d83a738bb8"} Oct 30 14:22:40.508331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.508293 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" event={"ID":"308ea75f-c712-4435-a0b7-5134ff7bd65c","Type":"ContainerStarted","Data":"ad5b8d7155ef306cb399c3096d119cbf5d0092aea05d63af1ef9a6ffcc403747"} Oct 30 14:22:40.508622 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.508594 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c22439-xvkcb\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:40.510509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.510487 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c22439-xvkcb\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:40.529782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.529755 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z28x\" (UniqueName: \"kubernetes.io/projected/fd3653da-8836-46e6-8c3b-dff86258aa8a-kube-api-access-2z28x\") pod \"maas-api-revoke-sub-d1c22439-xvkcb\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:40.580513 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.580458 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" podStartSLOduration=1.580445821 podStartE2EDuration="1.580445821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:40.580123295 +0000 UTC m=+956.205566671" watchObservedRunningTime="2026-10-30 14:22:40.580445821 +0000 UTC m=+956.205889193" Oct 30 14:22:40.587703 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.587678 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:40.900425 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:40.900384 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb"] Oct 30 14:22:40.901218 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:40.901186 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3653da_8836_46e6_8c3b_dff86258aa8a.slice/crio-f0e9083b1ee72ff1173aa56027e06ef19d9b46da0f20b663b8c07ff59435aa60 WatchSource:0}: Error finding container f0e9083b1ee72ff1173aa56027e06ef19d9b46da0f20b663b8c07ff59435aa60: Status 404 returned error can't find the container with id f0e9083b1ee72ff1173aa56027e06ef19d9b46da0f20b663b8c07ff59435aa60 Oct 30 14:22:41.516763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.516657 4968 generic.go:356] "Generic (PLEG): container finished" podID="308ea75f-c712-4435-a0b7-5134ff7bd65c" containerID="168b650286d672ff74a373896576f917cc44e7d78f27262be6fe91d83a738bb8" exitCode=0 Oct 30 14:22:41.516763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.516744 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" event={"ID":"308ea75f-c712-4435-a0b7-5134ff7bd65c","Type":"ContainerDied","Data":"168b650286d672ff74a373896576f917cc44e7d78f27262be6fe91d83a738bb8"} Oct 30 14:22:41.518490 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.518458 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" event={"ID":"fd3653da-8836-46e6-8c3b-dff86258aa8a","Type":"ContainerStarted","Data":"56c62654f40ccc4daf4f779664619928ef86ea0ba8d5e1e32446754885fa0748"} Oct 30 14:22:41.518626 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.518512 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" event={"ID":"fd3653da-8836-46e6-8c3b-dff86258aa8a","Type":"ContainerStarted","Data":"f0e9083b1ee72ff1173aa56027e06ef19d9b46da0f20b663b8c07ff59435aa60"} Oct 30 14:22:41.601383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.601321 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" podStartSLOduration=1.601302628 podStartE2EDuration="1.601302628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:41.598646346 +0000 UTC m=+957.224089720" watchObservedRunningTime="2026-10-30 14:22:41.601302628 +0000 UTC m=+957.226746003" Oct 30 14:22:41.704628 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.704604 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:41.831944 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.831915 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-xks-ca\") pod \"189c32fe-20bf-45e5-9aba-154f23d5e895\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " Oct 30 14:22:41.832138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.832056 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-openshift-ca\") pod \"189c32fe-20bf-45e5-9aba-154f23d5e895\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " Oct 30 14:22:41.832138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.832101 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189c32fe-20bf-45e5-9aba-154f23d5e895-kube-api-access-tq46p\" (UniqueName: \"kubernetes.io/projected/189c32fe-20bf-45e5-9aba-154f23d5e895-kube-api-access-tq46p\") pod \"189c32fe-20bf-45e5-9aba-154f23d5e895\" (UID: \"189c32fe-20bf-45e5-9aba-154f23d5e895\") " Oct 30 14:22:41.832403 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.832379 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-openshift-ca" pod "189c32fe-20bf-45e5-9aba-154f23d5e895" (UID: "189c32fe-20bf-45e5-9aba-154f23d5e895"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:41.832534 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.832522 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:41.834054 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.834031 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-xks-ca" pod "189c32fe-20bf-45e5-9aba-154f23d5e895" (UID: "189c32fe-20bf-45e5-9aba-154f23d5e895"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:41.834166 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.834097 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189c32fe-20bf-45e5-9aba-154f23d5e895-kube-api-access-tq46p" pod "189c32fe-20bf-45e5-9aba-154f23d5e895" (UID: "189c32fe-20bf-45e5-9aba-154f23d5e895"). InnerVolumeSpecName "kube-api-access-tq46p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:41.933967 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.933930 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq46p\" (UniqueName: \"kubernetes.io/projected/189c32fe-20bf-45e5-9aba-154f23d5e895-kube-api-access-tq46p\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:41.933967 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:41.933962 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189c32fe-20bf-45e5-9aba-154f23d5e895-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:42.528214 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.528181 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" Oct 30 14:22:42.528214 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.528193 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n" event={"ID":"189c32fe-20bf-45e5-9aba-154f23d5e895","Type":"ContainerDied","Data":"4720b1c3c27193cc78f634f05d97f0a955ca3554a1f1bb3612c83f07e906e449"} Oct 30 14:22:42.528757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.528247 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4720b1c3c27193cc78f634f05d97f0a955ca3554a1f1bb3612c83f07e906e449" Oct 30 14:22:42.530263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.530239 4968 generic.go:356] "Generic (PLEG): container finished" podID="fd3653da-8836-46e6-8c3b-dff86258aa8a" containerID="56c62654f40ccc4daf4f779664619928ef86ea0ba8d5e1e32446754885fa0748" exitCode=0 Oct 30 14:22:42.530378 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.530267 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" event={"ID":"fd3653da-8836-46e6-8c3b-dff86258aa8a","Type":"ContainerDied","Data":"56c62654f40ccc4daf4f779664619928ef86ea0ba8d5e1e32446754885fa0748"} Oct 30 14:22:42.754047 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.754019 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:42.945460 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.945428 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-xks-ca\") pod \"308ea75f-c712-4435-a0b7-5134ff7bd65c\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " Oct 30 14:22:42.945609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.945535 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-openshift-ca\") pod \"308ea75f-c712-4435-a0b7-5134ff7bd65c\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " Oct 30 14:22:42.945609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.945576 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308ea75f-c712-4435-a0b7-5134ff7bd65c-kube-api-access-md26z\" (UniqueName: \"kubernetes.io/projected/308ea75f-c712-4435-a0b7-5134ff7bd65c-kube-api-access-md26z\") pod \"308ea75f-c712-4435-a0b7-5134ff7bd65c\" (UID: \"308ea75f-c712-4435-a0b7-5134ff7bd65c\") " Oct 30 14:22:42.945938 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.945911 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-openshift-ca" pod "308ea75f-c712-4435-a0b7-5134ff7bd65c" (UID: "308ea75f-c712-4435-a0b7-5134ff7bd65c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:42.947594 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.947571 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-xks-ca" pod "308ea75f-c712-4435-a0b7-5134ff7bd65c" (UID: "308ea75f-c712-4435-a0b7-5134ff7bd65c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:42.947742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:42.947699 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308ea75f-c712-4435-a0b7-5134ff7bd65c-kube-api-access-md26z" pod "308ea75f-c712-4435-a0b7-5134ff7bd65c" (UID: "308ea75f-c712-4435-a0b7-5134ff7bd65c"). InnerVolumeSpecName "kube-api-access-md26z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:43.046966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:43.046934 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:43.046966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:43.046963 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308ea75f-c712-4435-a0b7-5134ff7bd65c-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:43.046966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:43.046973 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md26z\" (UniqueName: \"kubernetes.io/projected/308ea75f-c712-4435-a0b7-5134ff7bd65c-kube-api-access-md26z\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:43.540510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:43.540476 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" Oct 30 14:22:43.540510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:43.540488 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m" event={"ID":"308ea75f-c712-4435-a0b7-5134ff7bd65c","Type":"ContainerDied","Data":"ad5b8d7155ef306cb399c3096d119cbf5d0092aea05d63af1ef9a6ffcc403747"} Oct 30 14:22:43.540935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:43.540544 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5b8d7155ef306cb399c3096d119cbf5d0092aea05d63af1ef9a6ffcc403747" Oct 30 14:22:44.202739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.199251 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:44.254750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.254170 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-openshift-ca\") pod \"fd3653da-8836-46e6-8c3b-dff86258aa8a\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " Oct 30 14:22:44.260369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.260340 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-xks-ca\") pod \"fd3653da-8836-46e6-8c3b-dff86258aa8a\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " Oct 30 14:22:44.260688 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.260653 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3653da-8836-46e6-8c3b-dff86258aa8a-kube-api-access-2z28x\" (UniqueName: \"kubernetes.io/projected/fd3653da-8836-46e6-8c3b-dff86258aa8a-kube-api-access-2z28x\") pod \"fd3653da-8836-46e6-8c3b-dff86258aa8a\" (UID: \"fd3653da-8836-46e6-8c3b-dff86258aa8a\") " Oct 30 14:22:44.261215 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.260225 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-openshift-ca" pod "fd3653da-8836-46e6-8c3b-dff86258aa8a" (UID: "fd3653da-8836-46e6-8c3b-dff86258aa8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:44.262464 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.262440 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:44.445944 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.445533 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-xks-ca" pod "fd3653da-8836-46e6-8c3b-dff86258aa8a" (UID: "fd3653da-8836-46e6-8c3b-dff86258aa8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:44.450833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.450798 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3653da-8836-46e6-8c3b-dff86258aa8a-kube-api-access-2z28x" pod "fd3653da-8836-46e6-8c3b-dff86258aa8a" (UID: "fd3653da-8836-46e6-8c3b-dff86258aa8a"). InnerVolumeSpecName "kube-api-access-2z28x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:44.472186 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.472150 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z28x\" (UniqueName: \"kubernetes.io/projected/fd3653da-8836-46e6-8c3b-dff86258aa8a-kube-api-access-2z28x\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:44.472371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.472201 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3653da-8836-46e6-8c3b-dff86258aa8a-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:44.554737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.553794 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" event={"ID":"fd3653da-8836-46e6-8c3b-dff86258aa8a","Type":"ContainerDied","Data":"f0e9083b1ee72ff1173aa56027e06ef19d9b46da0f20b663b8c07ff59435aa60"} Oct 30 14:22:44.554737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.553857 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e9083b1ee72ff1173aa56027e06ef19d9b46da0f20b663b8c07ff59435aa60" Oct 30 14:22:44.554737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:44.553927 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb" Oct 30 14:22:45.892740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.891962 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j"] Oct 30 14:22:45.893240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.892791 4968 cpu_manager.go:414] "containerMap: removing container" podUID="189c32fe-20bf-45e5-9aba-154f23d5e895" containerName="revoke-keys" Oct 30 14:22:45.893240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.892810 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189c32fe-20bf-45e5-9aba-154f23d5e895" containerName="revoke-keys" Oct 30 14:22:45.893240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.892832 4968 cpu_manager.go:414] "containerMap: removing container" podUID="308ea75f-c712-4435-a0b7-5134ff7bd65c" containerName="revoke-keys" Oct 30 14:22:45.893240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.892841 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308ea75f-c712-4435-a0b7-5134ff7bd65c" containerName="revoke-keys" Oct 30 14:22:45.893240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.892892 4968 cpu_manager.go:414] "containerMap: removing container" podUID="fd3653da-8836-46e6-8c3b-dff86258aa8a" containerName="revoke-keys" Oct 30 14:22:45.893240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.892903 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3653da-8836-46e6-8c3b-dff86258aa8a" containerName="revoke-keys" Oct 30 14:22:45.893240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.893030 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3653da-8836-46e6-8c3b-dff86258aa8a" containerName="revoke-keys" Oct 30 14:22:45.893240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.893048 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="189c32fe-20bf-45e5-9aba-154f23d5e895" containerName="revoke-keys" Oct 30 14:22:45.893240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.893059 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="308ea75f-c712-4435-a0b7-5134ff7bd65c" containerName="revoke-keys" Oct 30 14:22:45.904822 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.900405 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:45.924803 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.924206 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-bf8266-openshift-default-dockercfg-9c8kq\"" Oct 30 14:22:45.937246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:45.937220 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j"] Oct 30 14:22:46.032765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.022852 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istiod-ca-cert\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.032765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.022929 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-data\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.032765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.022960 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-token\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.032765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.023003 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-envoy\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.032765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.023056 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-certs\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.032765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.023099 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-podinfo\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.032765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.023129 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9zrw\" (UniqueName: \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-kube-api-access-s9zrw\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.032765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.023168 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.032765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.023216 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-credential-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.317813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.309602 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istiod-ca-cert\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.317813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.309701 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-data\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.347740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.333318 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-token\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.347740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.333424 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-envoy\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.347740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.333503 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-certs\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.347740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.333562 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-podinfo\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.347740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.333602 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9zrw\" (UniqueName: \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-kube-api-access-s9zrw\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.347740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.333645 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.347740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.333709 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-credential-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.347740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.334326 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-credential-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.347740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.334624 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-data\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.347740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.337072 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-certs\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.349802 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.349775 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.469766 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.368708 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istiod-ca-cert\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.469766 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.374922 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-token\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.469766 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.451571 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-envoy\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.469766 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.465902 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9zrw\" (UniqueName: \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-kube-api-access-s9zrw\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.474742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.472402 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-podinfo\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:46.603743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:46.602947 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:47.208796 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:47.208698 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j"] Oct 30 14:22:47.266249 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:47.265945 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:22:47.266249 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:47.266066 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:22:47.266249 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:47.266112 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:22:47.594937 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:47.594895 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" event={"ID":"4c04a6cc-c66a-4671-beea-8df9ac3114cb","Type":"ContainerStarted","Data":"a2f946bdc08cd5e8300373cee2f65de4f0692c22abf3fb3fb134edf9a522b513"} Oct 30 14:22:47.595175 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:47.594961 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" event={"ID":"4c04a6cc-c66a-4671-beea-8df9ac3114cb","Type":"ContainerStarted","Data":"74db1cdb8d7fe996486658220465f4d38c03e4339a82c3f0122cd73c874f28a6"} Oct 30 14:22:47.604400 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:47.604341 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:47.650760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:47.649817 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" podStartSLOduration=2.649796256 podStartE2EDuration="2.649796256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:47.648435812 +0000 UTC m=+963.273879220" watchObservedRunningTime="2026-10-30 14:22:47.649796256 +0000 UTC m=+963.275239632" Oct 30 14:22:48.615930 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:48.615888 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:22:48.616397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:48.615946 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" podUID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:22:49.414875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.414839 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx"] Oct 30 14:22:49.419380 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.419347 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:49.439480 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.439441 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx"] Oct 30 14:22:49.445146 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.445114 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:22:49.445299 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.445147 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:22:49.445299 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.445158 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:22:49.445299 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.445225 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:22:49.531491 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.531433 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fd4567-p74vx\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:49.531687 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.531611 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fd4567-p74vx\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:49.531687 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.531656 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nlcq\" (UniqueName: \"kubernetes.io/projected/d975f97b-6681-4fe3-90a5-4d0b84590e70-kube-api-access-9nlcq\") pod \"maas-api-revoke-sub-16fd4567-p74vx\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:49.605154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.605118 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:22:49.605361 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.605169 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" podUID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:22:49.633302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.633270 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fd4567-p74vx\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:49.633706 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.633374 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fd4567-p74vx\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:49.633706 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.633416 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nlcq\" (UniqueName: \"kubernetes.io/projected/d975f97b-6681-4fe3-90a5-4d0b84590e70-kube-api-access-9nlcq\") pod \"maas-api-revoke-sub-16fd4567-p74vx\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:49.634077 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.634054 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fd4567-p74vx\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:49.635634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.635610 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fd4567-p74vx\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:49.655008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.654978 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nlcq\" (UniqueName: \"kubernetes.io/projected/d975f97b-6681-4fe3-90a5-4d0b84590e70-kube-api-access-9nlcq\") pod \"maas-api-revoke-sub-16fd4567-p74vx\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:49.744107 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:49.744018 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:50.070490 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.070465 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx"] Oct 30 14:22:50.072869 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:50.072835 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd975f97b_6681_4fe3_90a5_4d0b84590e70.slice/crio-6cc8c3121ec3387cf6f9b0619baef8b0bff4d71fdc9b30895543babfd762d328 WatchSource:0}: Error finding container 6cc8c3121ec3387cf6f9b0619baef8b0bff4d71fdc9b30895543babfd762d328: Status 404 returned error can't find the container with id 6cc8c3121ec3387cf6f9b0619baef8b0bff4d71fdc9b30895543babfd762d328 Oct 30 14:22:50.444856 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:22:50.444821 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:22:50.604110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.604066 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:22:50.604309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.604130 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" podUID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:22:50.618194 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.618161 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2"] Oct 30 14:22:50.622655 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.622634 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:50.632563 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.632538 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" event={"ID":"d975f97b-6681-4fe3-90a5-4d0b84590e70","Type":"ContainerStarted","Data":"b74bb2867d03a936c6a76d9ab696055368008569b405cd3d3d7037a2d16bef88"} Oct 30 14:22:50.632698 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.632578 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" event={"ID":"d975f97b-6681-4fe3-90a5-4d0b84590e70","Type":"ContainerStarted","Data":"6cc8c3121ec3387cf6f9b0619baef8b0bff4d71fdc9b30895543babfd762d328"} Oct 30 14:22:50.642416 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.642391 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2"] Oct 30 14:22:50.711998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.711892 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" podStartSLOduration=1.71187786 podStartE2EDuration="1.71187786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:50.710602374 +0000 UTC m=+966.336045747" watchObservedRunningTime="2026-10-30 14:22:50.71187786 +0000 UTC m=+966.337321232" Oct 30 14:22:50.752930 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.752889 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0226f211-dbff2\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:50.753099 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.753024 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjpq6\" (UniqueName: \"kubernetes.io/projected/ad09e501-9c7e-4554-b17d-569e84446ced-kube-api-access-fjpq6\") pod \"maas-api-revoke-sub-0226f211-dbff2\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:50.753140 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.753104 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0226f211-dbff2\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:50.854388 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.854352 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjpq6\" (UniqueName: \"kubernetes.io/projected/ad09e501-9c7e-4554-b17d-569e84446ced-kube-api-access-fjpq6\") pod \"maas-api-revoke-sub-0226f211-dbff2\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:50.854586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.854433 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0226f211-dbff2\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:50.854586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.854535 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0226f211-dbff2\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:50.855274 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.855253 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0226f211-dbff2\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:50.856883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.856861 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0226f211-dbff2\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:50.907489 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.907450 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjpq6\" (UniqueName: \"kubernetes.io/projected/ad09e501-9c7e-4554-b17d-569e84446ced-kube-api-access-fjpq6\") pod \"maas-api-revoke-sub-0226f211-dbff2\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:50.937402 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:50.937372 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:51.269969 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:51.269937 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2"] Oct 30 14:22:51.277764 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:51.277734 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad09e501_9c7e_4554_b17d_569e84446ced.slice/crio-f813496d387023cb3162acd31a7148a2517a9bdd4ffaa49d11ea5f33065cf8e8 WatchSource:0}: Error finding container f813496d387023cb3162acd31a7148a2517a9bdd4ffaa49d11ea5f33065cf8e8: Status 404 returned error can't find the container with id f813496d387023cb3162acd31a7148a2517a9bdd4ffaa49d11ea5f33065cf8e8 Oct 30 14:22:51.605466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:51.605023 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:22:51.605466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:51.605081 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" podUID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:22:51.640744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:51.640672 4968 generic.go:356] "Generic (PLEG): container finished" podID="d975f97b-6681-4fe3-90a5-4d0b84590e70" containerID="b74bb2867d03a936c6a76d9ab696055368008569b405cd3d3d7037a2d16bef88" exitCode=0 Oct 30 14:22:51.640744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:51.640709 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" event={"ID":"d975f97b-6681-4fe3-90a5-4d0b84590e70","Type":"ContainerDied","Data":"b74bb2867d03a936c6a76d9ab696055368008569b405cd3d3d7037a2d16bef88"} Oct 30 14:22:51.642423 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:51.642392 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" event={"ID":"ad09e501-9c7e-4554-b17d-569e84446ced","Type":"ContainerStarted","Data":"f813496d387023cb3162acd31a7148a2517a9bdd4ffaa49d11ea5f33065cf8e8"} Oct 30 14:22:52.604918 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:52.604864 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:22:52.605122 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:52.604933 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" podUID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:22:52.650308 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:52.650269 4968 generic.go:356] "Generic (PLEG): container finished" podID="ad09e501-9c7e-4554-b17d-569e84446ced" containerID="0bcd9a49c4ecf84e75269057e889b08412912654c2695c3df273b018c88d988b" exitCode=0 Oct 30 14:22:52.650747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:52.650308 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" event={"ID":"ad09e501-9c7e-4554-b17d-569e84446ced","Type":"ContainerDied","Data":"0bcd9a49c4ecf84e75269057e889b08412912654c2695c3df273b018c88d988b"} Oct 30 14:22:52.852983 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:52.852958 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:53.010230 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.010198 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-xks-ca\") pod \"d975f97b-6681-4fe3-90a5-4d0b84590e70\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " Oct 30 14:22:53.010407 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.010335 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-openshift-ca\") pod \"d975f97b-6681-4fe3-90a5-4d0b84590e70\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " Oct 30 14:22:53.010407 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.010402 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d975f97b-6681-4fe3-90a5-4d0b84590e70-kube-api-access-9nlcq\" (UniqueName: \"kubernetes.io/projected/d975f97b-6681-4fe3-90a5-4d0b84590e70-kube-api-access-9nlcq\") pod \"d975f97b-6681-4fe3-90a5-4d0b84590e70\" (UID: \"d975f97b-6681-4fe3-90a5-4d0b84590e70\") " Oct 30 14:22:53.010656 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.010606 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-openshift-ca" pod "d975f97b-6681-4fe3-90a5-4d0b84590e70" (UID: "d975f97b-6681-4fe3-90a5-4d0b84590e70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:53.012525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.012501 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d975f97b-6681-4fe3-90a5-4d0b84590e70-kube-api-access-9nlcq" pod "d975f97b-6681-4fe3-90a5-4d0b84590e70" (UID: "d975f97b-6681-4fe3-90a5-4d0b84590e70"). InnerVolumeSpecName "kube-api-access-9nlcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:53.012620 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.012588 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-xks-ca" pod "d975f97b-6681-4fe3-90a5-4d0b84590e70" (UID: "d975f97b-6681-4fe3-90a5-4d0b84590e70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:53.112216 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.112182 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:53.112216 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.112219 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nlcq\" (UniqueName: \"kubernetes.io/projected/d975f97b-6681-4fe3-90a5-4d0b84590e70-kube-api-access-9nlcq\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:53.112420 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.112229 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975f97b-6681-4fe3-90a5-4d0b84590e70-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:53.605557 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.605110 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:22:53.605557 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.605172 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" podUID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:22:53.660813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.660492 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" event={"ID":"d975f97b-6681-4fe3-90a5-4d0b84590e70","Type":"ContainerDied","Data":"6cc8c3121ec3387cf6f9b0619baef8b0bff4d71fdc9b30895543babfd762d328"} Oct 30 14:22:53.660813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.660560 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cc8c3121ec3387cf6f9b0619baef8b0bff4d71fdc9b30895543babfd762d328" Oct 30 14:22:53.661920 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.661899 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx" Oct 30 14:22:53.851997 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:53.851969 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:54.029373 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.029281 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad09e501-9c7e-4554-b17d-569e84446ced-kube-api-access-fjpq6\" (UniqueName: \"kubernetes.io/projected/ad09e501-9c7e-4554-b17d-569e84446ced-kube-api-access-fjpq6\") pod \"ad09e501-9c7e-4554-b17d-569e84446ced\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " Oct 30 14:22:54.029548 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.029441 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-openshift-ca\") pod \"ad09e501-9c7e-4554-b17d-569e84446ced\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " Oct 30 14:22:54.029548 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.029489 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-xks-ca\") pod \"ad09e501-9c7e-4554-b17d-569e84446ced\" (UID: \"ad09e501-9c7e-4554-b17d-569e84446ced\") " Oct 30 14:22:54.029915 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.029873 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-openshift-ca" pod "ad09e501-9c7e-4554-b17d-569e84446ced" (UID: "ad09e501-9c7e-4554-b17d-569e84446ced"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:54.031397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.031364 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad09e501-9c7e-4554-b17d-569e84446ced-kube-api-access-fjpq6" pod "ad09e501-9c7e-4554-b17d-569e84446ced" (UID: "ad09e501-9c7e-4554-b17d-569e84446ced"). InnerVolumeSpecName "kube-api-access-fjpq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:54.031498 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.031440 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-xks-ca" pod "ad09e501-9c7e-4554-b17d-569e84446ced" (UID: "ad09e501-9c7e-4554-b17d-569e84446ced"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:54.131242 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.131212 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:54.131242 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.131240 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad09e501-9c7e-4554-b17d-569e84446ced-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:54.131242 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.131250 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjpq6\" (UniqueName: \"kubernetes.io/projected/ad09e501-9c7e-4554-b17d-569e84446ced-kube-api-access-fjpq6\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:22:54.609902 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.609868 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:54.610277 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.610253 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:54.611073 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.611045 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:22:54.668661 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.668640 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" Oct 30 14:22:54.669043 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.668673 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2" event={"ID":"ad09e501-9c7e-4554-b17d-569e84446ced","Type":"ContainerDied","Data":"f813496d387023cb3162acd31a7148a2517a9bdd4ffaa49d11ea5f33065cf8e8"} Oct 30 14:22:54.669043 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.668729 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f813496d387023cb3162acd31a7148a2517a9bdd4ffaa49d11ea5f33065cf8e8" Oct 30 14:22:54.954400 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.954368 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd"] Oct 30 14:22:54.954991 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.954971 4968 cpu_manager.go:414] "containerMap: removing container" podUID="ad09e501-9c7e-4554-b17d-569e84446ced" containerName="revoke-keys" Oct 30 14:22:54.954991 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.954988 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad09e501-9c7e-4554-b17d-569e84446ced" containerName="revoke-keys" Oct 30 14:22:54.955158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.955007 4968 cpu_manager.go:414] "containerMap: removing container" podUID="d975f97b-6681-4fe3-90a5-4d0b84590e70" containerName="revoke-keys" Oct 30 14:22:54.955158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.955013 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d975f97b-6681-4fe3-90a5-4d0b84590e70" containerName="revoke-keys" Oct 30 14:22:54.955158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.955091 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="d975f97b-6681-4fe3-90a5-4d0b84590e70" containerName="revoke-keys" Oct 30 14:22:54.955158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.955105 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad09e501-9c7e-4554-b17d-569e84446ced" containerName="revoke-keys" Oct 30 14:22:54.959004 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.958983 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:54.978312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.978289 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:22:54.978312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.978305 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:22:54.978312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.978290 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:22:54.978474 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.978391 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:22:54.982436 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:54.982412 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd"] Oct 30 14:22:55.141448 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:55.141414 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f781596b-gwkqd\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:55.141625 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:55.141466 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f781596b-gwkqd\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:55.141625 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:55.141610 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vb4l\" (UniqueName: \"kubernetes.io/projected/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-kube-api-access-8vb4l\") pod \"maas-api-revoke-sub-f781596b-gwkqd\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:55.253778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:55.250941 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vb4l\" (UniqueName: \"kubernetes.io/projected/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-kube-api-access-8vb4l\") pod \"maas-api-revoke-sub-f781596b-gwkqd\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:55.253778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:55.251134 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f781596b-gwkqd\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:55.253778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:55.251201 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f781596b-gwkqd\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:55.253778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:55.252925 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f781596b-gwkqd\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:55.272335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:55.272293 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f781596b-gwkqd\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:55.276820 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:55.276763 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vb4l\" (UniqueName: \"kubernetes.io/projected/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-kube-api-access-8vb4l\") pod \"maas-api-revoke-sub-f781596b-gwkqd\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:55.574807 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:55.574693 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:55.885029 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:55.884997 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd"] Oct 30 14:22:55.887892 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:22:55.887861 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5a75e3_7932_4f81_bd14_c8b7d0e1166f.slice/crio-d457bdd865561107a2a933a3bbfb19e1ad13cc6163cc91f482bca08161108bdb WatchSource:0}: Error finding container d457bdd865561107a2a933a3bbfb19e1ad13cc6163cc91f482bca08161108bdb: Status 404 returned error can't find the container with id d457bdd865561107a2a933a3bbfb19e1ad13cc6163cc91f482bca08161108bdb Oct 30 14:22:56.687358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:56.687316 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" event={"ID":"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f","Type":"ContainerStarted","Data":"7f30f352d0efb187c0a41a2fa726a5ffd0712dae8cefaa84f46327dd8cd59f76"} Oct 30 14:22:56.687550 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:56.687385 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" event={"ID":"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f","Type":"ContainerStarted","Data":"d457bdd865561107a2a933a3bbfb19e1ad13cc6163cc91f482bca08161108bdb"} Oct 30 14:22:56.728418 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:56.728367 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" podStartSLOduration=2.728354444 podStartE2EDuration="2.728354444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:56.725551578 +0000 UTC m=+972.350994964" watchObservedRunningTime="2026-10-30 14:22:56.728354444 +0000 UTC m=+972.353797861" Oct 30 14:22:57.493518 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:57.493477 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:22:57.696883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:57.696853 4968 generic.go:356] "Generic (PLEG): container finished" podID="9e5a75e3-7932-4f81-bd14-c8b7d0e1166f" containerID="7f30f352d0efb187c0a41a2fa726a5ffd0712dae8cefaa84f46327dd8cd59f76" exitCode=0 Oct 30 14:22:57.697053 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:57.696941 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" event={"ID":"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f","Type":"ContainerDied","Data":"7f30f352d0efb187c0a41a2fa726a5ffd0712dae8cefaa84f46327dd8cd59f76"} Oct 30 14:22:59.407895 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:59.407861 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:22:59.815037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:59.814989 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-kube-api-access-8vb4l\" (UniqueName: \"kubernetes.io/projected/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-kube-api-access-8vb4l\") pod \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " Oct 30 14:22:59.815590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:59.815380 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-openshift-ca\") pod \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " Oct 30 14:22:59.815835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:59.815815 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-xks-ca\") pod \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\" (UID: \"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f\") " Oct 30 14:22:59.929414 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:59.929368 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-openshift-ca" pod "9e5a75e3-7932-4f81-bd14-c8b7d0e1166f" (UID: "9e5a75e3-7932-4f81-bd14-c8b7d0e1166f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:59.993844 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:22:59.993505 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:23:00.063262 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:00.059442 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-kube-api-access-8vb4l" pod "9e5a75e3-7932-4f81-bd14-c8b7d0e1166f" (UID: "9e5a75e3-7932-4f81-bd14-c8b7d0e1166f"). InnerVolumeSpecName "kube-api-access-8vb4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:00.063262 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:00.059558 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" podUID="99b00b7d-0f8b-41f7-b646-995797d34045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:23:00.074394 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:00.074302 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" event={"ID":"9e5a75e3-7932-4f81-bd14-c8b7d0e1166f","Type":"ContainerDied","Data":"d457bdd865561107a2a933a3bbfb19e1ad13cc6163cc91f482bca08161108bdb"} Oct 30 14:23:00.079765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:00.074655 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d457bdd865561107a2a933a3bbfb19e1ad13cc6163cc91f482bca08161108bdb" Oct 30 14:23:00.080485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:00.080436 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd" Oct 30 14:23:00.106741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:00.099948 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vb4l\" (UniqueName: \"kubernetes.io/projected/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-kube-api-access-8vb4l\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:23:00.106741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:00.100521 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-xks-ca" pod "9e5a75e3-7932-4f81-bd14-c8b7d0e1166f" (UID: "9e5a75e3-7932-4f81-bd14-c8b7d0e1166f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:23:00.207576 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:00.207534 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:23:01.455756 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.455118 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv"] Oct 30 14:23:01.460747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.460465 4968 cpu_manager.go:414] "containerMap: removing container" podUID="9e5a75e3-7932-4f81-bd14-c8b7d0e1166f" containerName="revoke-keys" Oct 30 14:23:01.460747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.460494 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5a75e3-7932-4f81-bd14-c8b7d0e1166f" containerName="revoke-keys" Oct 30 14:23:01.460747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.460655 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5a75e3-7932-4f81-bd14-c8b7d0e1166f" containerName="revoke-keys" Oct 30 14:23:01.470046 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.470017 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:01.477393 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.477360 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv"] Oct 30 14:23:01.496763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.492782 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:23:01.505244 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.505208 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:23:01.505626 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.505608 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:23:01.505808 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.505793 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:23:01.675318 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.675268 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493967ac-rkdnv\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:01.675790 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.675623 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-574px\" (UniqueName: \"kubernetes.io/projected/9f1a5ba3-86a2-4784-873e-e33d00fc5529-kube-api-access-574px\") pod \"maas-api-revoke-sub-493967ac-rkdnv\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:01.676524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.676494 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493967ac-rkdnv\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:01.782455 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.782353 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-574px\" (UniqueName: \"kubernetes.io/projected/9f1a5ba3-86a2-4784-873e-e33d00fc5529-kube-api-access-574px\") pod \"maas-api-revoke-sub-493967ac-rkdnv\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:01.782804 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.782782 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493967ac-rkdnv\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:01.783012 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.782994 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493967ac-rkdnv\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:01.784108 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.784081 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493967ac-rkdnv\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:01.792302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.792276 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493967ac-rkdnv\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:01.818780 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:01.808676 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-574px\" (UniqueName: \"kubernetes.io/projected/9f1a5ba3-86a2-4784-873e-e33d00fc5529-kube-api-access-574px\") pod \"maas-api-revoke-sub-493967ac-rkdnv\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:02.111867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.111836 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:02.678298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.678271 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv"] Oct 30 14:23:02.681484 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:23:02.681447 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1a5ba3_86a2_4784_873e_e33d00fc5529.slice/crio-7a70e3c107a7e62cae856501bcea66732ab7f01f0f698c87f00aada4510ed2a8 WatchSource:0}: Error finding container 7a70e3c107a7e62cae856501bcea66732ab7f01f0f698c87f00aada4510ed2a8: Status 404 returned error can't find the container with id 7a70e3c107a7e62cae856501bcea66732ab7f01f0f698c87f00aada4510ed2a8 Oct 30 14:23:02.685924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.685894 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd"] Oct 30 14:23:02.720954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.720931 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.756968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.756919 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-bf8266\"/\"openshift-service-ca.crt\"" Oct 30 14:23:02.757129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.757061 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bf8266\"/\"distinct-w6-bf8266-kserve-self-signed-certs\"" Oct 30 14:23:02.765265 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.757356 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bf8266\"/\"default-dockercfg-l8wgf\"" Oct 30 14:23:02.766246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.766223 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd"] Oct 30 14:23:02.766460 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.766443 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-bf8266\"/\"kube-root-ca.crt\"" Oct 30 14:23:02.772783 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.772758 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-tmp-dir\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.772912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.772890 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-model-cache\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.772971 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.772942 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-home\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.773020 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.772995 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-dshm\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.773069 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.773041 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwn9\" (UniqueName: \"kubernetes.io/projected/58305994-dcf4-44a4-afeb-b3e89cad85f1-kube-api-access-slwn9\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.773135 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.773071 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58305994-dcf4-44a4-afeb-b3e89cad85f1-tls-certs\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.874729 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.874674 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-dshm\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.874926 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.874776 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwn9\" (UniqueName: \"kubernetes.io/projected/58305994-dcf4-44a4-afeb-b3e89cad85f1-kube-api-access-slwn9\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.874926 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.874800 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58305994-dcf4-44a4-afeb-b3e89cad85f1-tls-certs\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.874926 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.874890 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-tmp-dir\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.875097 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.874995 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-model-cache\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.875097 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.875034 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-home\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.875431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.875396 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-model-cache\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.875431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.875417 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-home\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.875736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.875689 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-tmp-dir\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.877179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.877152 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-dshm\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.877614 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.877579 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58305994-dcf4-44a4-afeb-b3e89cad85f1-tls-certs\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:02.903445 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:02.903335 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwn9\" (UniqueName: \"kubernetes.io/projected/58305994-dcf4-44a4-afeb-b3e89cad85f1-kube-api-access-slwn9\") pod \"distinct-w6-bf8266-kserve-7b95b4754f-zdqmd\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:03.079827 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:03.079790 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:03.202965 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:03.202928 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" event={"ID":"9f1a5ba3-86a2-4784-873e-e33d00fc5529","Type":"ContainerStarted","Data":"7a70e3c107a7e62cae856501bcea66732ab7f01f0f698c87f00aada4510ed2a8"} Oct 30 14:23:03.445394 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:03.445370 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd"] Oct 30 14:23:03.447812 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:23:03.447786 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58305994_dcf4_44a4_afeb_b3e89cad85f1.slice/crio-515a627fdb47e4921756c229a9dca9707e1c49eab489ffcc8d8e9561d9310ce1 WatchSource:0}: Error finding container 515a627fdb47e4921756c229a9dca9707e1c49eab489ffcc8d8e9561d9310ce1: Status 404 returned error can't find the container with id 515a627fdb47e4921756c229a9dca9707e1c49eab489ffcc8d8e9561d9310ce1 Oct 30 14:23:04.211096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:04.211059 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" event={"ID":"58305994-dcf4-44a4-afeb-b3e89cad85f1","Type":"ContainerStarted","Data":"584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9"} Oct 30 14:23:04.211096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:04.211105 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" event={"ID":"58305994-dcf4-44a4-afeb-b3e89cad85f1","Type":"ContainerStarted","Data":"515a627fdb47e4921756c229a9dca9707e1c49eab489ffcc8d8e9561d9310ce1"} Oct 30 14:23:04.212980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:04.212953 4968 generic.go:356] "Generic (PLEG): container finished" podID="9f1a5ba3-86a2-4784-873e-e33d00fc5529" containerID="2fddbbfe21c8b8b88805c5787ce333ad91cef6c25e20d37a86b013113e3a8913" exitCode=0 Oct 30 14:23:04.213104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:04.212990 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" event={"ID":"9f1a5ba3-86a2-4784-873e-e33d00fc5529","Type":"ContainerDied","Data":"2fddbbfe21c8b8b88805c5787ce333ad91cef6c25e20d37a86b013113e3a8913"} Oct 30 14:23:04.252505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:04.252447 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" podStartSLOduration=2.252431655 podStartE2EDuration="2.252431655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:04.249772268 +0000 UTC m=+979.875215642" watchObservedRunningTime="2026-10-30 14:23:04.252431655 +0000 UTC m=+979.877875032" Oct 30 14:23:05.080236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:05.080204 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:05.373757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:05.373735 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:05.418934 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:05.418905 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-xks-ca\") pod \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " Oct 30 14:23:05.419130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:05.419014 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-openshift-ca\") pod \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " Oct 30 14:23:05.419130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:05.419075 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1a5ba3-86a2-4784-873e-e33d00fc5529-kube-api-access-574px\" (UniqueName: \"kubernetes.io/projected/9f1a5ba3-86a2-4784-873e-e33d00fc5529-kube-api-access-574px\") pod \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\" (UID: \"9f1a5ba3-86a2-4784-873e-e33d00fc5529\") " Oct 30 14:23:05.419423 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:05.419397 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-openshift-ca" pod "9f1a5ba3-86a2-4784-873e-e33d00fc5529" (UID: "9f1a5ba3-86a2-4784-873e-e33d00fc5529"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:23:05.421384 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:05.421347 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-xks-ca" pod "9f1a5ba3-86a2-4784-873e-e33d00fc5529" (UID: "9f1a5ba3-86a2-4784-873e-e33d00fc5529"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:23:05.422662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:05.422626 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1a5ba3-86a2-4784-873e-e33d00fc5529-kube-api-access-574px" pod "9f1a5ba3-86a2-4784-873e-e33d00fc5529" (UID: "9f1a5ba3-86a2-4784-873e-e33d00fc5529"). InnerVolumeSpecName "kube-api-access-574px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:05.520387 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:05.520356 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:23:05.520387 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:05.520383 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1a5ba3-86a2-4784-873e-e33d00fc5529-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:23:05.520387 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:05.520392 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-574px\" (UniqueName: \"kubernetes.io/projected/9f1a5ba3-86a2-4784-873e-e33d00fc5529-kube-api-access-574px\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:23:06.228109 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:06.228076 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" Oct 30 14:23:06.228291 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:06.228117 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv" event={"ID":"9f1a5ba3-86a2-4784-873e-e33d00fc5529","Type":"ContainerDied","Data":"7a70e3c107a7e62cae856501bcea66732ab7f01f0f698c87f00aada4510ed2a8"} Oct 30 14:23:06.228291 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:06.228175 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a70e3c107a7e62cae856501bcea66732ab7f01f0f698c87f00aada4510ed2a8" Oct 30 14:23:13.080438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:13.080405 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:13.093239 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:13.093216 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:13.395041 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:13.394946 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:23:20.452764 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:23:20.452707 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:23:20.890371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:20.890336 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:23:21.614364 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.614331 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4"] Oct 30 14:23:21.614919 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.614903 4968 cpu_manager.go:414] "containerMap: removing container" podUID="9f1a5ba3-86a2-4784-873e-e33d00fc5529" containerName="revoke-keys" Oct 30 14:23:21.614919 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.614920 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1a5ba3-86a2-4784-873e-e33d00fc5529" containerName="revoke-keys" Oct 30 14:23:21.615046 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.615036 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1a5ba3-86a2-4784-873e-e33d00fc5529" containerName="revoke-keys" Oct 30 14:23:21.620999 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.620968 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.658113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.658080 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4"] Oct 30 14:23:21.659215 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.659187 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a5ddf3\"/\"distinct-w2-a5ddf3-kserve-self-signed-certs\"" Oct 30 14:23:21.668977 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.668945 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-dshm\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.669139 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.669018 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-model-cache\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.669139 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.669088 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-tmp-dir\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.669259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.669204 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g55p\" (UniqueName: \"kubernetes.io/projected/9f8a5dab-f785-409d-bb57-0459127c0656-kube-api-access-8g55p\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.669346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.669329 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8a5dab-f785-409d-bb57-0459127c0656-tls-certs\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.669440 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.669413 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-home\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.770109 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.770079 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-model-cache\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.770310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.770138 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-tmp-dir\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.770310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.770171 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g55p\" (UniqueName: \"kubernetes.io/projected/9f8a5dab-f785-409d-bb57-0459127c0656-kube-api-access-8g55p\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.770310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.770257 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8a5dab-f785-409d-bb57-0459127c0656-tls-certs\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.770310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.770275 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-home\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.770553 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.770353 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-dshm\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.770553 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.770536 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-model-cache\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.771116 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.771090 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-tmp-dir\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.771818 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.771792 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-home\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.772966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.772939 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-dshm\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.774022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.773998 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8a5dab-f785-409d-bb57-0459127c0656-tls-certs\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.792980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.792947 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g55p\" (UniqueName: \"kubernetes.io/projected/9f8a5dab-f785-409d-bb57-0459127c0656-kube-api-access-8g55p\") pod \"distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:21.953256 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:21.953174 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:22.263621 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:22.263579 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4"] Oct 30 14:23:22.266095 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:23:22.266063 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f8a5dab_f785_409d_bb57_0459127c0656.slice/crio-cc92ff1e620fee705d68646e54dccfb29306a94964bd46619c67c1961e32ca9b WatchSource:0}: Error finding container cc92ff1e620fee705d68646e54dccfb29306a94964bd46619c67c1961e32ca9b: Status 404 returned error can't find the container with id cc92ff1e620fee705d68646e54dccfb29306a94964bd46619c67c1961e32ca9b Oct 30 14:23:22.473646 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:22.473603 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" event={"ID":"9f8a5dab-f785-409d-bb57-0459127c0656","Type":"ContainerStarted","Data":"5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919"} Oct 30 14:23:22.473867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:22.473656 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" event={"ID":"9f8a5dab-f785-409d-bb57-0459127c0656","Type":"ContainerStarted","Data":"cc92ff1e620fee705d68646e54dccfb29306a94964bd46619c67c1961e32ca9b"} Oct 30 14:23:22.516433 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:22.516317 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" podStartSLOduration=1.5162995339999998 podStartE2EDuration="1.516299534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:22.512734142 +0000 UTC m=+998.138177505" watchObservedRunningTime="2026-10-30 14:23:22.516299534 +0000 UTC m=+998.141742910" Oct 30 14:23:22.953839 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:22.953810 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:26.271674 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.271629 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl"] Oct 30 14:23:26.276246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.276223 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.299607 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.299563 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-24dfd7\"/\"distinct-w3-24dfd7-kserve-self-signed-certs\"" Oct 30 14:23:26.300428 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.300397 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl"] Oct 30 14:23:26.321608 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.321572 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-model-cache\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.321785 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.321628 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-tmp-dir\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.321785 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.321691 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-home\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.321920 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.321790 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47nfv\" (UniqueName: \"kubernetes.io/projected/48066b06-babd-49b6-961a-1bf34b8c7973-kube-api-access-47nfv\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.321920 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.321891 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-dshm\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.322037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.321956 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48066b06-babd-49b6-961a-1bf34b8c7973-tls-certs\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.423103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.423063 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48066b06-babd-49b6-961a-1bf34b8c7973-tls-certs\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.423307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.423154 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-model-cache\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.423307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.423184 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-tmp-dir\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.423307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.423200 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-home\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.423307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.423256 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nfv\" (UniqueName: \"kubernetes.io/projected/48066b06-babd-49b6-961a-1bf34b8c7973-kube-api-access-47nfv\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.423511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.423308 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-dshm\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.423625 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.423599 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-model-cache\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.423778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.423642 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-tmp-dir\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.423778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.423687 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-home\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.425747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.425699 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-dshm\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.426029 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.426008 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48066b06-babd-49b6-961a-1bf34b8c7973-tls-certs\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.451350 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.451316 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47nfv\" (UniqueName: \"kubernetes.io/projected/48066b06-babd-49b6-961a-1bf34b8c7973-kube-api-access-47nfv\") pod \"distinct-w3-24dfd7-kserve-784f985c5-c9bkl\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.599999 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.599957 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:26.954269 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:26.954241 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl"] Oct 30 14:23:26.956221 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:23:26.956197 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48066b06_babd_49b6_961a_1bf34b8c7973.slice/crio-10729ce8853ecd4c4da9a3f943a3a5fdfb70a9f148123e0a1ce6fa55b376b42f WatchSource:0}: Error finding container 10729ce8853ecd4c4da9a3f943a3a5fdfb70a9f148123e0a1ce6fa55b376b42f: Status 404 returned error can't find the container with id 10729ce8853ecd4c4da9a3f943a3a5fdfb70a9f148123e0a1ce6fa55b376b42f Oct 30 14:23:27.513755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:27.513687 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" event={"ID":"48066b06-babd-49b6-961a-1bf34b8c7973","Type":"ContainerStarted","Data":"1228f776cc0f6d045f4d62a9ae07c784e791a175f445b1398b3955c549f0c6aa"} Oct 30 14:23:27.514248 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:27.513766 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" event={"ID":"48066b06-babd-49b6-961a-1bf34b8c7973","Type":"ContainerStarted","Data":"10729ce8853ecd4c4da9a3f943a3a5fdfb70a9f148123e0a1ce6fa55b376b42f"} Oct 30 14:23:27.561738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:27.561665 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" podStartSLOduration=1.56165127 podStartE2EDuration="1.56165127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:27.558916078 +0000 UTC m=+1003.184359452" watchObservedRunningTime="2026-10-30 14:23:27.56165127 +0000 UTC m=+1003.187094640" Oct 30 14:23:27.600857 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:27.600826 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:31.534862 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.534822 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc"] Oct 30 14:23:31.540047 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.540010 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.567598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.567564 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc"] Oct 30 14:23:31.590036 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.589993 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bczx\" (UniqueName: \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-kube-api-access-5bczx\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.590201 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.590091 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52388676-c0d2-4229-86ac-ad7763946b31-istiod-ca-cert\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.590273 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.590200 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-istio-token\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.590273 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.590259 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.590398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.590290 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-certs\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.590398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.590313 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-data\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.590524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.590503 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-credential-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.590600 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.590536 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-envoy\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.590898 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.590863 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52388676-c0d2-4229-86ac-ad7763946b31-istio-podinfo\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.692280 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.692244 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-istio-token\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.692486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.692305 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.692486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.692332 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-certs\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.692486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.692354 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-data\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.692486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.692395 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-credential-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.692486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.692421 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-envoy\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.692486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.692482 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52388676-c0d2-4229-86ac-ad7763946b31-istio-podinfo\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.692836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.692566 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bczx\" (UniqueName: \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-kube-api-access-5bczx\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.692836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.692651 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52388676-c0d2-4229-86ac-ad7763946b31-istiod-ca-cert\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.693134 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.693105 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-data\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.693409 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.693384 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.693480 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.693437 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52388676-c0d2-4229-86ac-ad7763946b31-istiod-ca-cert\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.693692 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.693669 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-certs\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.693884 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.693774 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-credential-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.696438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.696412 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52388676-c0d2-4229-86ac-ad7763946b31-istio-podinfo\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.696647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.696625 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-envoy\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.720759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.715163 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-istio-token\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.720759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.715828 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bczx\" (UniqueName: \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-kube-api-access-5bczx\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.856513 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.856442 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:31.953505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.953450 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:31.968129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:31.967933 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:32.229757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:32.229697 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:32.230001 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:32.229976 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:32.230164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:32.230135 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:32.245995 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:32.245951 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc"] Oct 30 14:23:32.554166 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:32.554075 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" event={"ID":"52388676-c0d2-4229-86ac-ad7763946b31","Type":"ContainerStarted","Data":"338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47"} Oct 30 14:23:32.554166 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:32.554140 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" event={"ID":"52388676-c0d2-4229-86ac-ad7763946b31","Type":"ContainerStarted","Data":"1a6e072e73642a4d70822fe493dc550d35f25978ed885852d70dcad9be9bc8b7"} Oct 30 14:23:32.567065 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:32.567034 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:23:32.610518 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:32.610460 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" podStartSLOduration=1.610441832 podStartE2EDuration="1.610441832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:32.605812225 +0000 UTC m=+1008.231255598" watchObservedRunningTime="2026-10-30 14:23:32.610441832 +0000 UTC m=+1008.235885210" Oct 30 14:23:32.857885 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:32.857853 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:33.859070 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:33.859038 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 14:23:33.859466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:33.859091 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" podUID="52388676-c0d2-4229-86ac-ad7763946b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 14:23:34.857965 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:34.857919 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 14:23:34.858124 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:34.857990 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" podUID="52388676-c0d2-4229-86ac-ad7763946b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 14:23:35.858177 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:35.858137 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 14:23:35.858583 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:35.858196 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" podUID="52388676-c0d2-4229-86ac-ad7763946b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 14:23:36.607663 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:36.603878 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:36.630781 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:36.630750 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:36.857447 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:36.857415 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 14:23:36.857636 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:36.857468 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" podUID="52388676-c0d2-4229-86ac-ad7763946b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 14:23:37.620670 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:37.620629 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:23:37.857951 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:37.857914 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 14:23:37.858160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:37.857974 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" podUID="52388676-c0d2-4229-86ac-ad7763946b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 14:23:38.967222 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:38.967187 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:38.967764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:38.967745 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:38.969121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:38.969097 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:23:40.286742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:40.286698 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:23:44.917335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:44.917300 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8"] Oct 30 14:23:44.924751 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:44.924701 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:44.941692 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:44.941660 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-tmp-dir\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:44.942128 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:44.942106 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819c0707-1f66-400e-8704-c902586cae50-tls-certs\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:44.942455 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:44.942429 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-model-cache\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:44.942571 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:44.942478 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-home\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:44.942643 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:44.942609 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhz5c\" (UniqueName: \"kubernetes.io/projected/819c0707-1f66-400e-8704-c902586cae50-kube-api-access-xhz5c\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:44.942762 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:44.942741 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-dshm\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:44.951302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:44.950892 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bf8266\"/\"distinct-2-w6-bf8266-kserve-self-signed-certs\"" Oct 30 14:23:44.956304 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:44.956276 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8"] Oct 30 14:23:45.044445 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.044414 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-tmp-dir\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.044640 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.044470 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819c0707-1f66-400e-8704-c902586cae50-tls-certs\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.044640 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.044505 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-model-cache\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.044640 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.044530 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-home\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.044640 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.044573 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhz5c\" (UniqueName: \"kubernetes.io/projected/819c0707-1f66-400e-8704-c902586cae50-kube-api-access-xhz5c\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.044640 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.044620 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-dshm\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.044960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.044913 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-tmp-dir\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.045180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.045155 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-home\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.045321 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.045277 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-model-cache\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.046960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.046940 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-dshm\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.047351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.047332 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819c0707-1f66-400e-8704-c902586cae50-tls-certs\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.066951 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.066925 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhz5c\" (UniqueName: \"kubernetes.io/projected/819c0707-1f66-400e-8704-c902586cae50-kube-api-access-xhz5c\") pod \"distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.242977 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.242889 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:45.558432 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.558348 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8"] Oct 30 14:23:45.562279 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:23:45.562245 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod819c0707_1f66_400e_8704_c902586cae50.slice/crio-b56300d270b2f40120197e9f185a4dc1d9b90b65ad95f44f52991918409500ae WatchSource:0}: Error finding container b56300d270b2f40120197e9f185a4dc1d9b90b65ad95f44f52991918409500ae: Status 404 returned error can't find the container with id b56300d270b2f40120197e9f185a4dc1d9b90b65ad95f44f52991918409500ae Oct 30 14:23:45.680520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.680481 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" event={"ID":"819c0707-1f66-400e-8704-c902586cae50","Type":"ContainerStarted","Data":"db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3"} Oct 30 14:23:45.680750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.680531 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" event={"ID":"819c0707-1f66-400e-8704-c902586cae50","Type":"ContainerStarted","Data":"b56300d270b2f40120197e9f185a4dc1d9b90b65ad95f44f52991918409500ae"} Oct 30 14:23:45.725607 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:45.725553 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" podStartSLOduration=1.725540261 podStartE2EDuration="1.725540261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:45.722533208 +0000 UTC m=+1021.347976581" watchObservedRunningTime="2026-10-30 14:23:45.725540261 +0000 UTC m=+1021.350983643" Oct 30 14:23:46.243251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:46.243215 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:50.445501 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:23:50.445456 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:23:55.243221 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:55.243183 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:55.255824 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:55.255798 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:23:55.793456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:23:55.793429 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:24:01.060102 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.060029 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62"] Oct 30 14:24:01.065896 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.065873 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.103120 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.103085 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62"] Oct 30 14:24:01.192557 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.192519 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39235b85-adb6-477d-b069-64e29e05fd17-istio-podinfo\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.192557 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.192566 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2nl\" (UniqueName: \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-kube-api-access-4z2nl\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.192855 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.192681 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39235b85-adb6-477d-b069-64e29e05fd17-istiod-ca-cert\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.192855 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.192770 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.193211 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.193183 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-certs\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.193451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.193425 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-envoy\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.193562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.193520 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-data\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.193625 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.193585 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-credential-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.193680 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.193627 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-istio-token\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.294438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.294401 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39235b85-adb6-477d-b069-64e29e05fd17-istio-podinfo\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.294641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.294441 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2nl\" (UniqueName: \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-kube-api-access-4z2nl\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.294641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.294483 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39235b85-adb6-477d-b069-64e29e05fd17-istiod-ca-cert\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.294641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.294536 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.294641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.294630 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-certs\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.294894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.294710 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-envoy\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.294894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.294767 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-data\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.294894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.294806 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-credential-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.294894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.294854 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-istio-token\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.295091 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.295027 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.295176 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.295138 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-certs\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.295362 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.295332 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39235b85-adb6-477d-b069-64e29e05fd17-istiod-ca-cert\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.295501 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.295390 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-data\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.295620 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.295597 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-credential-socket\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.297009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.296984 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-envoy\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.297182 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.297164 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39235b85-adb6-477d-b069-64e29e05fd17-istio-podinfo\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.335440 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.335415 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2nl\" (UniqueName: \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-kube-api-access-4z2nl\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.339435 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.339396 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-istio-token\") pod \"e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.381170 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.379938 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:01.780318 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.780291 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62"] Oct 30 14:24:01.784162 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:24:01.784132 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39235b85_adb6_477d_b069_64e29e05fd17.slice/crio-750ef2b9e6cc63d69e673ddfd608fcc8cb7248e1d812b44a61dda508a4fc69a1 WatchSource:0}: Error finding container 750ef2b9e6cc63d69e673ddfd608fcc8cb7248e1d812b44a61dda508a4fc69a1: Status 404 returned error can't find the container with id 750ef2b9e6cc63d69e673ddfd608fcc8cb7248e1d812b44a61dda508a4fc69a1 Oct 30 14:24:01.787413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.787369 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:24:01.787507 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.787451 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:24:01.787507 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.787480 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:24:01.828019 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:01.827991 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" event={"ID":"39235b85-adb6-477d-b069-64e29e05fd17","Type":"ContainerStarted","Data":"750ef2b9e6cc63d69e673ddfd608fcc8cb7248e1d812b44a61dda508a4fc69a1"} Oct 30 14:24:02.846410 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:02.846364 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" event={"ID":"39235b85-adb6-477d-b069-64e29e05fd17","Type":"ContainerStarted","Data":"2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c"} Oct 30 14:24:02.908366 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:02.908304 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" podStartSLOduration=1.9082812260000002 podStartE2EDuration="1.908281226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:02.903560779 +0000 UTC m=+1038.529004158" watchObservedRunningTime="2026-10-30 14:24:02.908281226 +0000 UTC m=+1038.533724602" Oct 30 14:24:03.400740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:03.399789 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:03.409393 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:03.409359 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 14:24:03.409545 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:03.409428 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" podUID="39235b85-adb6-477d-b069-64e29e05fd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 14:24:04.403740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:04.403113 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 14:24:04.403740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:04.403213 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" podUID="39235b85-adb6-477d-b069-64e29e05fd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 14:24:05.396742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:05.385062 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 14:24:05.396742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:05.385121 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" podUID="39235b85-adb6-477d-b069-64e29e05fd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 14:24:06.383061 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:06.383025 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 14:24:06.383639 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:06.383605 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" podUID="39235b85-adb6-477d-b069-64e29e05fd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 14:24:07.393739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:07.385045 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 14:24:07.393739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:07.385120 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" podUID="39235b85-adb6-477d-b069-64e29e05fd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 14:24:08.381444 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:08.381400 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 14:24:08.381654 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:08.381481 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" podUID="39235b85-adb6-477d-b069-64e29e05fd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 14:24:09.386135 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:09.386104 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:09.386668 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:09.386647 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:09.387502 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:09.387472 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:24:09.820216 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:09.820128 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:24:12.248345 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.247372 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6"] Oct 30 14:24:12.255795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.255755 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.266140 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.266107 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6"] Oct 30 14:24:12.270750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.270700 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-24dfd7\"/\"distinct-2-w3-24dfd7-kserve-self-signed-certs\"" Oct 30 14:24:12.331646 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.331013 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-dshm\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.331646 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.331094 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-home\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.331646 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.331168 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tmp-dir\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.331646 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.331244 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tls-certs\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.331646 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.331286 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-model-cache\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.331646 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.331359 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8867z\" (UniqueName: \"kubernetes.io/projected/554e6fd9-60f6-49bb-a2a4-ef944cc98136-kube-api-access-8867z\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.432983 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.432939 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-dshm\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.433162 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.433012 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-home\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.433162 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.433080 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tmp-dir\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.433162 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.433146 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tls-certs\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.433349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.433182 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-model-cache\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.433349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.433245 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8867z\" (UniqueName: \"kubernetes.io/projected/554e6fd9-60f6-49bb-a2a4-ef944cc98136-kube-api-access-8867z\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.433531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.433496 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-home\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.433531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.433527 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tmp-dir\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.433696 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.433571 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-model-cache\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.435741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.435699 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-dshm\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.435741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.435739 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tls-certs\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.455011 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.454985 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8867z\" (UniqueName: \"kubernetes.io/projected/554e6fd9-60f6-49bb-a2a4-ef944cc98136-kube-api-access-8867z\") pod \"distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.581345 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.579793 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:12.917507 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:12.917479 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6"] Oct 30 14:24:12.919970 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:24:12.919942 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod554e6fd9_60f6_49bb_a2a4_ef944cc98136.slice/crio-7613dcdbfb397ad35b53736ab2d2a5e512484ec08c02c493bce9c365da489688 WatchSource:0}: Error finding container 7613dcdbfb397ad35b53736ab2d2a5e512484ec08c02c493bce9c365da489688: Status 404 returned error can't find the container with id 7613dcdbfb397ad35b53736ab2d2a5e512484ec08c02c493bce9c365da489688 Oct 30 14:24:13.932614 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:13.932576 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" event={"ID":"554e6fd9-60f6-49bb-a2a4-ef944cc98136","Type":"ContainerStarted","Data":"e2cbd1e7c6870e96da50293d85f4dd63e3e76a13e8f7d65f138d87a9a8bdc427"} Oct 30 14:24:13.933131 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:13.932627 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" event={"ID":"554e6fd9-60f6-49bb-a2a4-ef944cc98136","Type":"ContainerStarted","Data":"7613dcdbfb397ad35b53736ab2d2a5e512484ec08c02c493bce9c365da489688"} Oct 30 14:24:13.974233 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:13.974183 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" podStartSLOduration=1.974169334 podStartE2EDuration="1.974169334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:13.97176875 +0000 UTC m=+1049.597212123" watchObservedRunningTime="2026-10-30 14:24:13.974169334 +0000 UTC m=+1049.599612741" Oct 30 14:24:14.581223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:14.581186 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:20.444061 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:24:20.444019 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:24:22.580692 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:22.580657 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:22.598772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:22.598745 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:23.020777 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:23.020680 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:24:25.905776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:25.905708 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv"] Oct 30 14:24:25.938049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:25.938014 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:25.960186 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:25.956408 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:24:25.970278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:25.970251 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:24:25.970562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:25.970541 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:24:25.970802 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:25.970785 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:24:25.972681 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:25.972531 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv"] Oct 30 14:24:26.067483 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:26.067447 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e55eaf-wnrcv\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:26.067638 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:26.067554 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e55eaf-wnrcv\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:26.067691 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:26.067655 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8bqn\" (UniqueName: \"kubernetes.io/projected/9e0e3219-8050-48de-98b3-0962851a502b-kube-api-access-p8bqn\") pod \"maas-api-revoke-sub-99e55eaf-wnrcv\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:26.169145 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:26.169054 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e55eaf-wnrcv\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:26.169145 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:26.169111 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e55eaf-wnrcv\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:26.169390 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:26.169183 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8bqn\" (UniqueName: \"kubernetes.io/projected/9e0e3219-8050-48de-98b3-0962851a502b-kube-api-access-p8bqn\") pod \"maas-api-revoke-sub-99e55eaf-wnrcv\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:26.170062 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:26.170033 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e55eaf-wnrcv\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:26.171868 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:26.171834 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e55eaf-wnrcv\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:26.191342 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:26.191311 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8bqn\" (UniqueName: \"kubernetes.io/projected/9e0e3219-8050-48de-98b3-0962851a502b-kube-api-access-p8bqn\") pod \"maas-api-revoke-sub-99e55eaf-wnrcv\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:26.275880 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:26.275845 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:26.587833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:26.587801 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv"] Oct 30 14:24:26.588186 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:24:26.588164 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0e3219_8050_48de_98b3_0962851a502b.slice/crio-aed13b52f2994c230235c314b0c9367bd7dbba018f01272a5065e918d67368f4 WatchSource:0}: Error finding container aed13b52f2994c230235c314b0c9367bd7dbba018f01272a5065e918d67368f4: Status 404 returned error can't find the container with id aed13b52f2994c230235c314b0c9367bd7dbba018f01272a5065e918d67368f4 Oct 30 14:24:27.118464 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.118425 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" event={"ID":"9e0e3219-8050-48de-98b3-0962851a502b","Type":"ContainerStarted","Data":"1257645e89bf8f5d369f50d2e1bd22472d3939b7b10ffbfda41197ea8a6cef08"} Oct 30 14:24:27.118988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.118503 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" event={"ID":"9e0e3219-8050-48de-98b3-0962851a502b","Type":"ContainerStarted","Data":"aed13b52f2994c230235c314b0c9367bd7dbba018f01272a5065e918d67368f4"} Oct 30 14:24:27.177132 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.177009 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" podStartSLOduration=2.176994585 podStartE2EDuration="2.176994585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:27.174593094 +0000 UTC m=+1062.800036466" watchObservedRunningTime="2026-10-30 14:24:27.176994585 +0000 UTC m=+1062.802437958" Oct 30 14:24:27.347704 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.347674 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk"] Oct 30 14:24:27.351634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.351615 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:27.372622 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.372593 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk"] Oct 30 14:24:27.486444 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.486349 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b5696b-7fqbk\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:27.486444 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.486411 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b5696b-7fqbk\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:27.486661 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.486545 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnt5h\" (UniqueName: \"kubernetes.io/projected/d14f6418-d532-46fb-aabe-988b8516284c-kube-api-access-fnt5h\") pod \"maas-api-revoke-sub-c1b5696b-7fqbk\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:27.587897 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.587865 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnt5h\" (UniqueName: \"kubernetes.io/projected/d14f6418-d532-46fb-aabe-988b8516284c-kube-api-access-fnt5h\") pod \"maas-api-revoke-sub-c1b5696b-7fqbk\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:27.588075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.587986 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b5696b-7fqbk\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:27.588075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.588035 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b5696b-7fqbk\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:27.588820 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.588791 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b5696b-7fqbk\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:27.590690 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.590662 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b5696b-7fqbk\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:27.610048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.610022 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnt5h\" (UniqueName: \"kubernetes.io/projected/d14f6418-d532-46fb-aabe-988b8516284c-kube-api-access-fnt5h\") pod \"maas-api-revoke-sub-c1b5696b-7fqbk\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:27.666118 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.666094 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:27.978411 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:27.978385 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk"] Oct 30 14:24:27.979455 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:24:27.979424 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14f6418_d532_46fb_aabe_988b8516284c.slice/crio-4b81e7c6347ded5bdfbab9513f818994f8d393648ff47240b3954c1893305b74 WatchSource:0}: Error finding container 4b81e7c6347ded5bdfbab9513f818994f8d393648ff47240b3954c1893305b74: Status 404 returned error can't find the container with id 4b81e7c6347ded5bdfbab9513f818994f8d393648ff47240b3954c1893305b74 Oct 30 14:24:28.128961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:28.128923 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" event={"ID":"d14f6418-d532-46fb-aabe-988b8516284c","Type":"ContainerStarted","Data":"4b81e7c6347ded5bdfbab9513f818994f8d393648ff47240b3954c1893305b74"} Oct 30 14:24:28.130573 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:28.130541 4968 generic.go:356] "Generic (PLEG): container finished" podID="9e0e3219-8050-48de-98b3-0962851a502b" containerID="1257645e89bf8f5d369f50d2e1bd22472d3939b7b10ffbfda41197ea8a6cef08" exitCode=0 Oct 30 14:24:28.130706 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:28.130600 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" event={"ID":"9e0e3219-8050-48de-98b3-0962851a502b","Type":"ContainerDied","Data":"1257645e89bf8f5d369f50d2e1bd22472d3939b7b10ffbfda41197ea8a6cef08"} Oct 30 14:24:29.138645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.138607 4968 generic.go:356] "Generic (PLEG): container finished" podID="d14f6418-d532-46fb-aabe-988b8516284c" containerID="7d04954eb49a9a38d013442930db7929e5f909722aa989ff301dd5effe74ef14" exitCode=0 Oct 30 14:24:29.139037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.138696 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" event={"ID":"d14f6418-d532-46fb-aabe-988b8516284c","Type":"ContainerDied","Data":"7d04954eb49a9a38d013442930db7929e5f909722aa989ff301dd5effe74ef14"} Oct 30 14:24:29.295641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.295614 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:29.328136 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.328091 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-openshift-ca\") pod \"9e0e3219-8050-48de-98b3-0962851a502b\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " Oct 30 14:24:29.328337 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.328287 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-xks-ca\") pod \"9e0e3219-8050-48de-98b3-0962851a502b\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " Oct 30 14:24:29.328450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.328433 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0e3219-8050-48de-98b3-0962851a502b-kube-api-access-p8bqn\" (UniqueName: \"kubernetes.io/projected/9e0e3219-8050-48de-98b3-0962851a502b-kube-api-access-p8bqn\") pod \"9e0e3219-8050-48de-98b3-0962851a502b\" (UID: \"9e0e3219-8050-48de-98b3-0962851a502b\") " Oct 30 14:24:29.328516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.328452 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-openshift-ca" pod "9e0e3219-8050-48de-98b3-0962851a502b" (UID: "9e0e3219-8050-48de-98b3-0962851a502b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:24:29.328920 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.328891 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:29.330527 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.330506 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-xks-ca" pod "9e0e3219-8050-48de-98b3-0962851a502b" (UID: "9e0e3219-8050-48de-98b3-0962851a502b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:24:29.330624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.330529 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0e3219-8050-48de-98b3-0962851a502b-kube-api-access-p8bqn" pod "9e0e3219-8050-48de-98b3-0962851a502b" (UID: "9e0e3219-8050-48de-98b3-0962851a502b"). InnerVolumeSpecName "kube-api-access-p8bqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:29.430286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.430245 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8bqn\" (UniqueName: \"kubernetes.io/projected/9e0e3219-8050-48de-98b3-0962851a502b-kube-api-access-p8bqn\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:29.430286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:29.430284 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0e3219-8050-48de-98b3-0962851a502b-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:30.146295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.146263 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" Oct 30 14:24:30.146687 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.146291 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv" event={"ID":"9e0e3219-8050-48de-98b3-0962851a502b","Type":"ContainerDied","Data":"aed13b52f2994c230235c314b0c9367bd7dbba018f01272a5065e918d67368f4"} Oct 30 14:24:30.146687 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.146341 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aed13b52f2994c230235c314b0c9367bd7dbba018f01272a5065e918d67368f4" Oct 30 14:24:30.315519 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.315490 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:30.340916 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.340890 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14f6418-d532-46fb-aabe-988b8516284c-kube-api-access-fnt5h\" (UniqueName: \"kubernetes.io/projected/d14f6418-d532-46fb-aabe-988b8516284c-kube-api-access-fnt5h\") pod \"d14f6418-d532-46fb-aabe-988b8516284c\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " Oct 30 14:24:30.341056 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.340921 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-xks-ca\") pod \"d14f6418-d532-46fb-aabe-988b8516284c\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " Oct 30 14:24:30.341133 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.341064 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-openshift-ca\") pod \"d14f6418-d532-46fb-aabe-988b8516284c\" (UID: \"d14f6418-d532-46fb-aabe-988b8516284c\") " Oct 30 14:24:30.341390 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.341365 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-openshift-ca" pod "d14f6418-d532-46fb-aabe-988b8516284c" (UID: "d14f6418-d532-46fb-aabe-988b8516284c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:24:30.341526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.341513 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:30.342990 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.342969 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-xks-ca" pod "d14f6418-d532-46fb-aabe-988b8516284c" (UID: "d14f6418-d532-46fb-aabe-988b8516284c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:24:30.343067 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.343014 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14f6418-d532-46fb-aabe-988b8516284c-kube-api-access-fnt5h" pod "d14f6418-d532-46fb-aabe-988b8516284c" (UID: "d14f6418-d532-46fb-aabe-988b8516284c"). InnerVolumeSpecName "kube-api-access-fnt5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:30.475835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.475707 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnt5h\" (UniqueName: \"kubernetes.io/projected/d14f6418-d532-46fb-aabe-988b8516284c-kube-api-access-fnt5h\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:30.475835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:30.475769 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14f6418-d532-46fb-aabe-988b8516284c-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:31.157286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:31.157241 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" event={"ID":"d14f6418-d532-46fb-aabe-988b8516284c","Type":"ContainerDied","Data":"4b81e7c6347ded5bdfbab9513f818994f8d393648ff47240b3954c1893305b74"} Oct 30 14:24:31.157782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:31.157343 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b81e7c6347ded5bdfbab9513f818994f8d393648ff47240b3954c1893305b74" Oct 30 14:24:31.157782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:31.157435 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk" Oct 30 14:24:39.691032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.690985 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx"] Oct 30 14:24:39.691605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.691589 4968 cpu_manager.go:414] "containerMap: removing container" podUID="9e0e3219-8050-48de-98b3-0962851a502b" containerName="revoke-keys" Oct 30 14:24:39.691605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.691605 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0e3219-8050-48de-98b3-0962851a502b" containerName="revoke-keys" Oct 30 14:24:39.691770 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.691637 4968 cpu_manager.go:414] "containerMap: removing container" podUID="d14f6418-d532-46fb-aabe-988b8516284c" containerName="revoke-keys" Oct 30 14:24:39.691770 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.691642 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14f6418-d532-46fb-aabe-988b8516284c" containerName="revoke-keys" Oct 30 14:24:39.691885 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.691783 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0e3219-8050-48de-98b3-0962851a502b" containerName="revoke-keys" Oct 30 14:24:39.691885 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.691797 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14f6418-d532-46fb-aabe-988b8516284c" containerName="revoke-keys" Oct 30 14:24:39.696664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.696640 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:39.710869 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.710848 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:24:39.723595 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.723562 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:24:39.723762 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.723607 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:24:39.727831 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.727801 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx"] Oct 30 14:24:39.732444 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.732418 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm"] Oct 30 14:24:39.747757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.744533 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:39.762765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.758834 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm"] Oct 30 14:24:39.792871 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.792839 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc029af9-h5qpx\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:39.792999 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.792892 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779ml\" (UniqueName: \"kubernetes.io/projected/5e55be46-5a85-4a9d-a82e-587bef35635b-kube-api-access-779ml\") pod \"maas-api-revoke-sub-dc029af9-h5qpx\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:39.793065 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.793014 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc029af9-h5qpx\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:39.810893 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.810870 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:24:39.893950 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.893921 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-739e8215-h5tvm\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:39.894102 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.893974 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc029af9-h5qpx\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:39.894102 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.894057 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779ml\" (UniqueName: \"kubernetes.io/projected/5e55be46-5a85-4a9d-a82e-587bef35635b-kube-api-access-779ml\") pod \"maas-api-revoke-sub-dc029af9-h5qpx\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:39.894228 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.894209 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc029af9-h5qpx\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:39.894422 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.894401 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vql8\" (UniqueName: \"kubernetes.io/projected/c665c064-6a70-4808-9c35-cbf48f29f0b7-kube-api-access-5vql8\") pod \"maas-api-revoke-sub-739e8215-h5tvm\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:39.894549 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.894456 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-739e8215-h5tvm\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:39.895012 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.894980 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc029af9-h5qpx\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:39.896549 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.896523 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc029af9-h5qpx\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:39.916517 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.916491 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779ml\" (UniqueName: \"kubernetes.io/projected/5e55be46-5a85-4a9d-a82e-587bef35635b-kube-api-access-779ml\") pod \"maas-api-revoke-sub-dc029af9-h5qpx\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:39.995970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.995893 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vql8\" (UniqueName: \"kubernetes.io/projected/c665c064-6a70-4808-9c35-cbf48f29f0b7-kube-api-access-5vql8\") pod \"maas-api-revoke-sub-739e8215-h5tvm\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:39.995970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.995939 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-739e8215-h5tvm\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:39.996132 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.995978 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-739e8215-h5tvm\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:39.996735 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.996685 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-739e8215-h5tvm\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:39.998402 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:39.998376 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-739e8215-h5tvm\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:40.012896 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:40.012869 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:40.018424 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:40.018397 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vql8\" (UniqueName: \"kubernetes.io/projected/c665c064-6a70-4808-9c35-cbf48f29f0b7-kube-api-access-5vql8\") pod \"maas-api-revoke-sub-739e8215-h5tvm\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:40.065271 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:40.065240 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:40.334526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:40.334496 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx"] Oct 30 14:24:40.336336 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:24:40.336306 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e55be46_5a85_4a9d_a82e_587bef35635b.slice/crio-db30846c7591d64032291e0f9045972c0bf143e01595442ef438ae3466c298d8 WatchSource:0}: Error finding container db30846c7591d64032291e0f9045972c0bf143e01595442ef438ae3466c298d8: Status 404 returned error can't find the container with id db30846c7591d64032291e0f9045972c0bf143e01595442ef438ae3466c298d8 Oct 30 14:24:40.446334 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:40.446306 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm"] Oct 30 14:24:40.448317 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:24:40.448285 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc665c064_6a70_4808_9c35_cbf48f29f0b7.slice/crio-33735c5dd024259862bca13d645e52ffc2705f2d9468acdcc028540beec88ba4 WatchSource:0}: Error finding container 33735c5dd024259862bca13d645e52ffc2705f2d9468acdcc028540beec88ba4: Status 404 returned error can't find the container with id 33735c5dd024259862bca13d645e52ffc2705f2d9468acdcc028540beec88ba4 Oct 30 14:24:41.236332 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:41.236301 4968 generic.go:356] "Generic (PLEG): container finished" podID="5e55be46-5a85-4a9d-a82e-587bef35635b" containerID="8712ac0cccc2f922178f90ac51906db55ec6b1dac2f5715798d72b7cfbb5e6b7" exitCode=0 Oct 30 14:24:41.236688 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:41.236377 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" event={"ID":"5e55be46-5a85-4a9d-a82e-587bef35635b","Type":"ContainerDied","Data":"8712ac0cccc2f922178f90ac51906db55ec6b1dac2f5715798d72b7cfbb5e6b7"} Oct 30 14:24:41.236688 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:41.236430 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" event={"ID":"5e55be46-5a85-4a9d-a82e-587bef35635b","Type":"ContainerStarted","Data":"db30846c7591d64032291e0f9045972c0bf143e01595442ef438ae3466c298d8"} Oct 30 14:24:41.238040 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:41.238020 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" event={"ID":"c665c064-6a70-4808-9c35-cbf48f29f0b7","Type":"ContainerStarted","Data":"1b488bcb9d4f6899569f354a16ad616bed9d6233d5237e6a360c30f04005aa28"} Oct 30 14:24:41.238147 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:41.238063 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" event={"ID":"c665c064-6a70-4808-9c35-cbf48f29f0b7","Type":"ContainerStarted","Data":"33735c5dd024259862bca13d645e52ffc2705f2d9468acdcc028540beec88ba4"} Oct 30 14:24:41.310106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:41.310049 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" podStartSLOduration=2.31003633 podStartE2EDuration="2.31003633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:41.309562852 +0000 UTC m=+1076.935006228" watchObservedRunningTime="2026-10-30 14:24:41.31003633 +0000 UTC m=+1076.935479703" Oct 30 14:24:42.125912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.125873 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:24:42.284736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.284532 4968 generic.go:356] "Generic (PLEG): container finished" podID="c665c064-6a70-4808-9c35-cbf48f29f0b7" containerID="1b488bcb9d4f6899569f354a16ad616bed9d6233d5237e6a360c30f04005aa28" exitCode=0 Oct 30 14:24:42.294054 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.285249 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" event={"ID":"c665c064-6a70-4808-9c35-cbf48f29f0b7","Type":"ContainerDied","Data":"1b488bcb9d4f6899569f354a16ad616bed9d6233d5237e6a360c30f04005aa28"} Oct 30 14:24:42.467235 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.467205 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:42.636812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.636776 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-xks-ca\") pod \"5e55be46-5a85-4a9d-a82e-587bef35635b\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " Oct 30 14:24:42.637021 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.636838 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-openshift-ca\") pod \"5e55be46-5a85-4a9d-a82e-587bef35635b\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " Oct 30 14:24:42.637021 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.636902 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e55be46-5a85-4a9d-a82e-587bef35635b-kube-api-access-779ml\" (UniqueName: \"kubernetes.io/projected/5e55be46-5a85-4a9d-a82e-587bef35635b-kube-api-access-779ml\") pod \"5e55be46-5a85-4a9d-a82e-587bef35635b\" (UID: \"5e55be46-5a85-4a9d-a82e-587bef35635b\") " Oct 30 14:24:42.637192 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.637162 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-openshift-ca" pod "5e55be46-5a85-4a9d-a82e-587bef35635b" (UID: "5e55be46-5a85-4a9d-a82e-587bef35635b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:24:42.637605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.637578 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:42.639098 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.639071 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-xks-ca" pod "5e55be46-5a85-4a9d-a82e-587bef35635b" (UID: "5e55be46-5a85-4a9d-a82e-587bef35635b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:24:42.639319 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.639299 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e55be46-5a85-4a9d-a82e-587bef35635b-kube-api-access-779ml" pod "5e55be46-5a85-4a9d-a82e-587bef35635b" (UID: "5e55be46-5a85-4a9d-a82e-587bef35635b"). InnerVolumeSpecName "kube-api-access-779ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:42.739527 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.739434 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e55be46-5a85-4a9d-a82e-587bef35635b-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:42.739527 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:42.739481 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-779ml\" (UniqueName: \"kubernetes.io/projected/5e55be46-5a85-4a9d-a82e-587bef35635b-kube-api-access-779ml\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:43.323601 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.323523 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" Oct 30 14:24:43.329331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.328430 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx" event={"ID":"5e55be46-5a85-4a9d-a82e-587bef35635b","Type":"ContainerDied","Data":"db30846c7591d64032291e0f9045972c0bf143e01595442ef438ae3466c298d8"} Oct 30 14:24:43.329331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.328497 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db30846c7591d64032291e0f9045972c0bf143e01595442ef438ae3466c298d8" Oct 30 14:24:43.667555 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.667524 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:43.775505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.775469 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c665c064-6a70-4808-9c35-cbf48f29f0b7-kube-api-access-5vql8\" (UniqueName: \"kubernetes.io/projected/c665c064-6a70-4808-9c35-cbf48f29f0b7-kube-api-access-5vql8\") pod \"c665c064-6a70-4808-9c35-cbf48f29f0b7\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " Oct 30 14:24:43.775674 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.775521 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-openshift-ca\") pod \"c665c064-6a70-4808-9c35-cbf48f29f0b7\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " Oct 30 14:24:43.775815 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.775793 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-xks-ca\") pod \"c665c064-6a70-4808-9c35-cbf48f29f0b7\" (UID: \"c665c064-6a70-4808-9c35-cbf48f29f0b7\") " Oct 30 14:24:43.776212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.776183 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-openshift-ca" pod "c665c064-6a70-4808-9c35-cbf48f29f0b7" (UID: "c665c064-6a70-4808-9c35-cbf48f29f0b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:24:43.778236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.778211 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c665c064-6a70-4808-9c35-cbf48f29f0b7-kube-api-access-5vql8" pod "c665c064-6a70-4808-9c35-cbf48f29f0b7" (UID: "c665c064-6a70-4808-9c35-cbf48f29f0b7"). InnerVolumeSpecName "kube-api-access-5vql8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:43.778562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.778541 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-xks-ca" pod "c665c064-6a70-4808-9c35-cbf48f29f0b7" (UID: "c665c064-6a70-4808-9c35-cbf48f29f0b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:24:43.878175 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.878082 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vql8\" (UniqueName: \"kubernetes.io/projected/c665c064-6a70-4808-9c35-cbf48f29f0b7-kube-api-access-5vql8\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:43.878175 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.878126 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:43.878175 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:43.878144 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c665c064-6a70-4808-9c35-cbf48f29f0b7-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:44.348209 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:44.348174 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" event={"ID":"c665c064-6a70-4808-9c35-cbf48f29f0b7","Type":"ContainerDied","Data":"33735c5dd024259862bca13d645e52ffc2705f2d9468acdcc028540beec88ba4"} Oct 30 14:24:44.348940 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:44.348920 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33735c5dd024259862bca13d645e52ffc2705f2d9468acdcc028540beec88ba4" Oct 30 14:24:44.349162 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:44.349148 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm" Oct 30 14:24:45.735430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.735392 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd"] Oct 30 14:24:45.736217 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.736200 4968 cpu_manager.go:414] "containerMap: removing container" podUID="c665c064-6a70-4808-9c35-cbf48f29f0b7" containerName="revoke-keys" Oct 30 14:24:45.736272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.736220 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c665c064-6a70-4808-9c35-cbf48f29f0b7" containerName="revoke-keys" Oct 30 14:24:45.736272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.736266 4968 cpu_manager.go:414] "containerMap: removing container" podUID="5e55be46-5a85-4a9d-a82e-587bef35635b" containerName="revoke-keys" Oct 30 14:24:45.736340 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.736275 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e55be46-5a85-4a9d-a82e-587bef35635b" containerName="revoke-keys" Oct 30 14:24:45.736416 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.736405 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="c665c064-6a70-4808-9c35-cbf48f29f0b7" containerName="revoke-keys" Oct 30 14:24:45.736450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.736426 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e55be46-5a85-4a9d-a82e-587bef35635b" containerName="revoke-keys" Oct 30 14:24:45.740566 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.740547 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:45.744444 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.744418 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hng\" (UniqueName: \"kubernetes.io/projected/bfd35601-9586-49af-9064-29963233e94c-kube-api-access-t5hng\") pod \"maas-api-revoke-sub-7fe8a064-tjrvd\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:45.744554 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.744453 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd35601-9586-49af-9064-29963233e94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe8a064-tjrvd\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:45.744629 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.744563 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd35601-9586-49af-9064-29963233e94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe8a064-tjrvd\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:45.774441 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.774412 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:24:45.774441 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.774419 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:24:45.774441 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.774426 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:24:45.774696 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.774458 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:24:45.780711 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.780681 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd"] Oct 30 14:24:45.845624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.845590 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hng\" (UniqueName: \"kubernetes.io/projected/bfd35601-9586-49af-9064-29963233e94c-kube-api-access-t5hng\") pod \"maas-api-revoke-sub-7fe8a064-tjrvd\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:45.845624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.845627 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd35601-9586-49af-9064-29963233e94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe8a064-tjrvd\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:45.845842 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.845678 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd35601-9586-49af-9064-29963233e94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe8a064-tjrvd\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:45.846508 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.846476 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd35601-9586-49af-9064-29963233e94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe8a064-tjrvd\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:45.848140 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.848115 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd35601-9586-49af-9064-29963233e94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe8a064-tjrvd\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:45.879057 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:45.879027 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hng\" (UniqueName: \"kubernetes.io/projected/bfd35601-9586-49af-9064-29963233e94c-kube-api-access-t5hng\") pod \"maas-api-revoke-sub-7fe8a064-tjrvd\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:46.055477 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:46.055385 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:46.365281 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:46.365251 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd"] Oct 30 14:24:46.368253 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:24:46.368223 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd35601_9586_49af_9064_29963233e94c.slice/crio-59e102150f799204c279095b0a81ed40b7f9c241a3b7fe7a425d4724dcd63cb0 WatchSource:0}: Error finding container 59e102150f799204c279095b0a81ed40b7f9c241a3b7fe7a425d4724dcd63cb0: Status 404 returned error can't find the container with id 59e102150f799204c279095b0a81ed40b7f9c241a3b7fe7a425d4724dcd63cb0 Oct 30 14:24:47.373652 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:47.373615 4968 generic.go:356] "Generic (PLEG): container finished" podID="bfd35601-9586-49af-9064-29963233e94c" containerID="1443a6a7f0f1a81fec5ff2eece37dc3049ad50c7d6c5523acafeac2b1add46bc" exitCode=0 Oct 30 14:24:47.374079 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:47.373704 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" event={"ID":"bfd35601-9586-49af-9064-29963233e94c","Type":"ContainerDied","Data":"1443a6a7f0f1a81fec5ff2eece37dc3049ad50c7d6c5523acafeac2b1add46bc"} Oct 30 14:24:47.374079 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:47.373811 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" event={"ID":"bfd35601-9586-49af-9064-29963233e94c","Type":"ContainerStarted","Data":"59e102150f799204c279095b0a81ed40b7f9c241a3b7fe7a425d4724dcd63cb0"} Oct 30 14:24:48.534821 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:48.534794 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:48.580114 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:48.580080 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd35601-9586-49af-9064-29963233e94c-kube-api-access-t5hng\" (UniqueName: \"kubernetes.io/projected/bfd35601-9586-49af-9064-29963233e94c-kube-api-access-t5hng\") pod \"bfd35601-9586-49af-9064-29963233e94c\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " Oct 30 14:24:48.580286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:48.580254 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfd35601-9586-49af-9064-29963233e94c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd35601-9586-49af-9064-29963233e94c-maas-api-xks-ca\") pod \"bfd35601-9586-49af-9064-29963233e94c\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " Oct 30 14:24:48.580443 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:48.580422 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd35601-9586-49af-9064-29963233e94c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd35601-9586-49af-9064-29963233e94c-maas-api-openshift-ca\") pod \"bfd35601-9586-49af-9064-29963233e94c\" (UID: \"bfd35601-9586-49af-9064-29963233e94c\") " Oct 30 14:24:48.581876 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:48.581830 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd35601-9586-49af-9064-29963233e94c-maas-api-openshift-ca" pod "bfd35601-9586-49af-9064-29963233e94c" (UID: "bfd35601-9586-49af-9064-29963233e94c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:24:48.591794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:48.591634 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd35601-9586-49af-9064-29963233e94c-kube-api-access-t5hng" pod "bfd35601-9586-49af-9064-29963233e94c" (UID: "bfd35601-9586-49af-9064-29963233e94c"). InnerVolumeSpecName "kube-api-access-t5hng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:48.591953 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:48.591915 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfd35601-9586-49af-9064-29963233e94c-maas-api-xks-ca" pod "bfd35601-9586-49af-9064-29963233e94c" (UID: "bfd35601-9586-49af-9064-29963233e94c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:24:48.682842 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:48.682770 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd35601-9586-49af-9064-29963233e94c-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:48.682842 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:48.682795 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd35601-9586-49af-9064-29963233e94c-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:48.682842 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:48.682807 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5hng\" (UniqueName: \"kubernetes.io/projected/bfd35601-9586-49af-9064-29963233e94c-kube-api-access-t5hng\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:49.414638 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:49.414599 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" event={"ID":"bfd35601-9586-49af-9064-29963233e94c","Type":"ContainerDied","Data":"59e102150f799204c279095b0a81ed40b7f9c241a3b7fe7a425d4724dcd63cb0"} Oct 30 14:24:49.415181 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:49.415142 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e102150f799204c279095b0a81ed40b7f9c241a3b7fe7a425d4724dcd63cb0" Oct 30 14:24:49.415361 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:49.414946 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd" Oct 30 14:24:50.445415 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:24:50.445380 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:24:54.812735 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.811866 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7"] Oct 30 14:24:54.813294 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.812986 4968 cpu_manager.go:414] "containerMap: removing container" podUID="bfd35601-9586-49af-9064-29963233e94c" containerName="revoke-keys" Oct 30 14:24:54.813294 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.813015 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd35601-9586-49af-9064-29963233e94c" containerName="revoke-keys" Oct 30 14:24:54.813294 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.813266 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd35601-9586-49af-9064-29963233e94c" containerName="revoke-keys" Oct 30 14:24:54.830746 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.830265 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:54.836791 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.836112 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7"] Oct 30 14:24:54.855794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.850079 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:24:54.855794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.850345 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:24:54.860202 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.860121 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:24:54.860202 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.860181 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:24:54.880871 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.880837 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7859n\" (UniqueName: \"kubernetes.io/projected/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-kube-api-access-7859n\") pod \"maas-api-revoke-sub-f779cb64-zjdn7\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:54.881066 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.881043 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f779cb64-zjdn7\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:54.881143 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.881080 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f779cb64-zjdn7\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:54.982040 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.982005 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f779cb64-zjdn7\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:54.982183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.982047 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f779cb64-zjdn7\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:54.982183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.982101 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7859n\" (UniqueName: \"kubernetes.io/projected/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-kube-api-access-7859n\") pod \"maas-api-revoke-sub-f779cb64-zjdn7\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:54.982955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.982926 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f779cb64-zjdn7\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:54.984539 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:54.984512 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f779cb64-zjdn7\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:55.005643 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:55.005610 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7859n\" (UniqueName: \"kubernetes.io/projected/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-kube-api-access-7859n\") pod \"maas-api-revoke-sub-f779cb64-zjdn7\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:55.168265 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:55.166925 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:55.604878 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:55.604776 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7"] Oct 30 14:24:55.608549 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:24:55.608506 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec31a55_ad49_4fc6_8e6a_32205bceec8b.slice/crio-afce962126a7b37b7700f7a337001e79fdee80c8af7be18a393bac73d5173349 WatchSource:0}: Error finding container afce962126a7b37b7700f7a337001e79fdee80c8af7be18a393bac73d5173349: Status 404 returned error can't find the container with id afce962126a7b37b7700f7a337001e79fdee80c8af7be18a393bac73d5173349 Oct 30 14:24:56.512773 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:56.504772 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" event={"ID":"4ec31a55-ad49-4fc6-8e6a-32205bceec8b","Type":"ContainerStarted","Data":"9dd3e7175c8146b0effe3445f5a2fdae65fff0bc3b11fd7a8a99fd58cf9af47b"} Oct 30 14:24:56.512773 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:56.504843 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" event={"ID":"4ec31a55-ad49-4fc6-8e6a-32205bceec8b","Type":"ContainerStarted","Data":"afce962126a7b37b7700f7a337001e79fdee80c8af7be18a393bac73d5173349"} Oct 30 14:24:56.572742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:56.572613 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" podStartSLOduration=2.572593312 podStartE2EDuration="2.572593312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:56.554062629 +0000 UTC m=+1092.179506003" watchObservedRunningTime="2026-10-30 14:24:56.572593312 +0000 UTC m=+1092.198036688" Oct 30 14:24:57.515206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:57.515170 4968 generic.go:356] "Generic (PLEG): container finished" podID="4ec31a55-ad49-4fc6-8e6a-32205bceec8b" containerID="9dd3e7175c8146b0effe3445f5a2fdae65fff0bc3b11fd7a8a99fd58cf9af47b" exitCode=0 Oct 30 14:24:57.515649 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:57.515260 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" event={"ID":"4ec31a55-ad49-4fc6-8e6a-32205bceec8b","Type":"ContainerDied","Data":"9dd3e7175c8146b0effe3445f5a2fdae65fff0bc3b11fd7a8a99fd58cf9af47b"} Oct 30 14:24:58.026216 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.026179 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb"] Oct 30 14:24:58.030422 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.030405 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:24:58.050385 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.050358 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb"] Oct 30 14:24:58.068847 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.068817 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqktz\" (UniqueName: \"kubernetes.io/projected/3bbba234-6f5d-46ba-beae-72844a22252e-kube-api-access-mqktz\") pod \"maas-api-revoke-sub-fd59e5c6-xzsjb\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:24:58.069004 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.068977 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd59e5c6-xzsjb\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:24:58.069200 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.069180 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd59e5c6-xzsjb\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:24:58.170035 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.170001 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd59e5c6-xzsjb\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:24:58.170201 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.170084 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqktz\" (UniqueName: \"kubernetes.io/projected/3bbba234-6f5d-46ba-beae-72844a22252e-kube-api-access-mqktz\") pod \"maas-api-revoke-sub-fd59e5c6-xzsjb\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:24:58.170201 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.170149 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd59e5c6-xzsjb\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:24:58.170694 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.170670 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd59e5c6-xzsjb\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:24:58.172427 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.172402 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd59e5c6-xzsjb\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:24:58.192243 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.192213 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqktz\" (UniqueName: \"kubernetes.io/projected/3bbba234-6f5d-46ba-beae-72844a22252e-kube-api-access-mqktz\") pod \"maas-api-revoke-sub-fd59e5c6-xzsjb\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:24:58.345856 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.345562 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:24:58.671935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.671911 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:58.689966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.689342 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb"] Oct 30 14:24:58.689966 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:24:58.689683 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbba234_6f5d_46ba_beae_72844a22252e.slice/crio-aa6c78530cd312c9c981b997059b5166c9eca0c90a07d7238ef7be6b613edcd1 WatchSource:0}: Error finding container aa6c78530cd312c9c981b997059b5166c9eca0c90a07d7238ef7be6b613edcd1: Status 404 returned error can't find the container with id aa6c78530cd312c9c981b997059b5166c9eca0c90a07d7238ef7be6b613edcd1 Oct 30 14:24:58.776520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.776489 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-kube-api-access-7859n\" (UniqueName: \"kubernetes.io/projected/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-kube-api-access-7859n\") pod \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " Oct 30 14:24:58.776711 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.776695 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-openshift-ca\") pod \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " Oct 30 14:24:58.776810 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.776790 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-xks-ca\") pod \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\" (UID: \"4ec31a55-ad49-4fc6-8e6a-32205bceec8b\") " Oct 30 14:24:58.777084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.777056 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-openshift-ca" pod "4ec31a55-ad49-4fc6-8e6a-32205bceec8b" (UID: "4ec31a55-ad49-4fc6-8e6a-32205bceec8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:24:58.777264 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.777254 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:58.779209 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.779184 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-xks-ca" pod "4ec31a55-ad49-4fc6-8e6a-32205bceec8b" (UID: "4ec31a55-ad49-4fc6-8e6a-32205bceec8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:24:58.779329 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.779210 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-kube-api-access-7859n" pod "4ec31a55-ad49-4fc6-8e6a-32205bceec8b" (UID: "4ec31a55-ad49-4fc6-8e6a-32205bceec8b"). InnerVolumeSpecName "kube-api-access-7859n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:58.879032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.878999 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:58.879032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:58.879030 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7859n\" (UniqueName: \"kubernetes.io/projected/4ec31a55-ad49-4fc6-8e6a-32205bceec8b-kube-api-access-7859n\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:24:59.533781 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:59.533747 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" Oct 30 14:24:59.533940 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:59.533745 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7" event={"ID":"4ec31a55-ad49-4fc6-8e6a-32205bceec8b","Type":"ContainerDied","Data":"afce962126a7b37b7700f7a337001e79fdee80c8af7be18a393bac73d5173349"} Oct 30 14:24:59.533940 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:59.533886 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afce962126a7b37b7700f7a337001e79fdee80c8af7be18a393bac73d5173349" Oct 30 14:24:59.535581 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:59.535548 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" event={"ID":"3bbba234-6f5d-46ba-beae-72844a22252e","Type":"ContainerStarted","Data":"8cb07c92c9f1604157a125f3a327dbf44691e0b171cfdc3a71206c768d9a7d37"} Oct 30 14:24:59.535705 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:59.535603 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" event={"ID":"3bbba234-6f5d-46ba-beae-72844a22252e","Type":"ContainerStarted","Data":"aa6c78530cd312c9c981b997059b5166c9eca0c90a07d7238ef7be6b613edcd1"} Oct 30 14:24:59.583725 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:24:59.583657 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" podStartSLOduration=1.583644372 podStartE2EDuration="1.583644372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:59.572468253 +0000 UTC m=+1095.197911626" watchObservedRunningTime="2026-10-30 14:24:59.583644372 +0000 UTC m=+1095.209087744" Oct 30 14:25:00.170561 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.170527 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv"] Oct 30 14:25:00.171134 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.171118 4968 cpu_manager.go:414] "containerMap: removing container" podUID="4ec31a55-ad49-4fc6-8e6a-32205bceec8b" containerName="revoke-keys" Oct 30 14:25:00.171196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.171135 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec31a55-ad49-4fc6-8e6a-32205bceec8b" containerName="revoke-keys" Oct 30 14:25:00.171248 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.171239 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec31a55-ad49-4fc6-8e6a-32205bceec8b" containerName="revoke-keys" Oct 30 14:25:00.182483 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.182446 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:00.199340 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.199310 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv"] Oct 30 14:25:00.304170 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.304135 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqdmq\" (UniqueName: \"kubernetes.io/projected/afef9040-cff0-40c0-b50a-9f0aea368e8d-kube-api-access-mqdmq\") pod \"maas-api-revoke-sub-fa00f5b9-6rnkv\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:00.304354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.304225 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa00f5b9-6rnkv\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:00.304434 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.304408 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa00f5b9-6rnkv\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:00.405929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.405899 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqdmq\" (UniqueName: \"kubernetes.io/projected/afef9040-cff0-40c0-b50a-9f0aea368e8d-kube-api-access-mqdmq\") pod \"maas-api-revoke-sub-fa00f5b9-6rnkv\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:00.406102 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.405988 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa00f5b9-6rnkv\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:00.406102 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.406080 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa00f5b9-6rnkv\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:00.406638 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.406619 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa00f5b9-6rnkv\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:00.408569 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.408543 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa00f5b9-6rnkv\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:00.428629 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.428566 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqdmq\" (UniqueName: \"kubernetes.io/projected/afef9040-cff0-40c0-b50a-9f0aea368e8d-kube-api-access-mqdmq\") pod \"maas-api-revoke-sub-fa00f5b9-6rnkv\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:00.496856 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.496815 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:00.545496 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.545464 4968 generic.go:356] "Generic (PLEG): container finished" podID="3bbba234-6f5d-46ba-beae-72844a22252e" containerID="8cb07c92c9f1604157a125f3a327dbf44691e0b171cfdc3a71206c768d9a7d37" exitCode=0 Oct 30 14:25:00.545655 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.545536 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" event={"ID":"3bbba234-6f5d-46ba-beae-72844a22252e","Type":"ContainerDied","Data":"8cb07c92c9f1604157a125f3a327dbf44691e0b171cfdc3a71206c768d9a7d37"} Oct 30 14:25:00.806458 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:00.806435 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv"] Oct 30 14:25:00.808739 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:00.808694 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafef9040_cff0_40c0_b50a_9f0aea368e8d.slice/crio-2a99e0c5828028d9f50fc4952f6a40b7d9f5fbe443e3b981524be811b8983bae WatchSource:0}: Error finding container 2a99e0c5828028d9f50fc4952f6a40b7d9f5fbe443e3b981524be811b8983bae: Status 404 returned error can't find the container with id 2a99e0c5828028d9f50fc4952f6a40b7d9f5fbe443e3b981524be811b8983bae Oct 30 14:25:01.365838 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.365754 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl"] Oct 30 14:25:01.370251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.370229 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.406160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.406129 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl"] Oct 30 14:25:01.428005 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.427973 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqv6m\" (UniqueName: \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-kube-api-access-sqv6m\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.428154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.428030 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istiod-ca-cert\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.428154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.428110 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-envoy\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.428277 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.428172 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-podinfo\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.428277 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.428243 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-token\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.428391 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.428280 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-credential-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.428391 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.428319 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.428391 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.428347 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-certs\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.428542 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.428391 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-data\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.529698 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.529663 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-podinfo\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.529917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.529769 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-token\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.529917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.529806 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-credential-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.529917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.529843 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.529917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.529875 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-certs\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.529917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.529914 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-data\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.530205 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.530007 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqv6m\" (UniqueName: \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-kube-api-access-sqv6m\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.530205 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.530049 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istiod-ca-cert\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.530205 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.530129 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-envoy\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.530645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.530411 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.530645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.530518 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-credential-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.530645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.530610 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-certs\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.530847 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.530671 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-data\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.531018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.530992 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istiod-ca-cert\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.532369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.532343 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-envoy\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.532452 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.532421 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-podinfo\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.552726 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.552683 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqv6m\" (UniqueName: \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-kube-api-access-sqv6m\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.552896 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.552877 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-token\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.556925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.556897 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" event={"ID":"afef9040-cff0-40c0-b50a-9f0aea368e8d","Type":"ContainerStarted","Data":"8dba7b8e855cc93aca1521cb66a61128774a50a976f41d15a51c92ebc83ddf4b"} Oct 30 14:25:01.557030 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.556955 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" event={"ID":"afef9040-cff0-40c0-b50a-9f0aea368e8d","Type":"ContainerStarted","Data":"2a99e0c5828028d9f50fc4952f6a40b7d9f5fbe443e3b981524be811b8983bae"} Oct 30 14:25:01.686181 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.685799 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:01.770846 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.770806 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:25:01.825187 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.824942 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" podStartSLOduration=1.8249217610000001 podStartE2EDuration="1.824921761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:01.640062556 +0000 UTC m=+1097.265505932" watchObservedRunningTime="2026-10-30 14:25:01.824921761 +0000 UTC m=+1097.450365137" Oct 30 14:25:01.832251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.832206 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb"] Oct 30 14:25:01.833123 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.833103 4968 cpu_manager.go:414] "containerMap: removing container" podUID="3bbba234-6f5d-46ba-beae-72844a22252e" containerName="revoke-keys" Oct 30 14:25:01.833256 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.833246 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bbba234-6f5d-46ba-beae-72844a22252e" containerName="revoke-keys" Oct 30 14:25:01.833488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.833477 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bbba234-6f5d-46ba-beae-72844a22252e" containerName="revoke-keys" Oct 30 14:25:01.837058 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.837039 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:01.861922 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.861891 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-xks-ca\") pod \"3bbba234-6f5d-46ba-beae-72844a22252e\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " Oct 30 14:25:01.862125 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.861941 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-openshift-ca\") pod \"3bbba234-6f5d-46ba-beae-72844a22252e\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " Oct 30 14:25:01.862125 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.862036 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bbba234-6f5d-46ba-beae-72844a22252e-kube-api-access-mqktz\" (UniqueName: \"kubernetes.io/projected/3bbba234-6f5d-46ba-beae-72844a22252e-kube-api-access-mqktz\") pod \"3bbba234-6f5d-46ba-beae-72844a22252e\" (UID: \"3bbba234-6f5d-46ba-beae-72844a22252e\") " Oct 30 14:25:01.863810 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.863493 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-openshift-ca" pod "3bbba234-6f5d-46ba-beae-72844a22252e" (UID: "3bbba234-6f5d-46ba-beae-72844a22252e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:01.866421 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.866378 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bbba234-6f5d-46ba-beae-72844a22252e-kube-api-access-mqktz" pod "3bbba234-6f5d-46ba-beae-72844a22252e" (UID: "3bbba234-6f5d-46ba-beae-72844a22252e"). InnerVolumeSpecName "kube-api-access-mqktz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:01.867985 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.867958 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-xks-ca" pod "3bbba234-6f5d-46ba-beae-72844a22252e" (UID: "3bbba234-6f5d-46ba-beae-72844a22252e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:01.874541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.874504 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb"] Oct 30 14:25:01.963502 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.963404 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-data\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:01.963502 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.963456 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istiod-ca-cert\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:01.963773 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.963576 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-credential-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:01.963773 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.963700 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hpv4\" (UniqueName: \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-kube-api-access-7hpv4\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:01.963862 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.963789 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-envoy\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:01.963951 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.963932 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:01.964023 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.964005 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-token\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:01.964070 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.964055 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-podinfo\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:01.964162 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.964138 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-certs\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:01.964282 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.964260 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqktz\" (UniqueName: \"kubernetes.io/projected/3bbba234-6f5d-46ba-beae-72844a22252e-kube-api-access-mqktz\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:01.964455 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.964294 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:01.964455 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:01.964310 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bbba234-6f5d-46ba-beae-72844a22252e-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:02.000941 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.000906 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5"] Oct 30 14:25:02.005341 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.005312 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.030966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.030938 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5"] Oct 30 14:25:02.059477 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.059442 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:02.059591 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.059508 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:02.059591 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.059546 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:02.063281 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.063252 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl"] Oct 30 14:25:02.066472 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.066432 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-token\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.066574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.066508 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-podinfo\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.066574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.066547 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-certs\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.066703 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.066584 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-data\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.066703 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.066616 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istiod-ca-cert\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.066703 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.066666 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-credential-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.066878 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.066753 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hpv4\" (UniqueName: \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-kube-api-access-7hpv4\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.066878 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.066797 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-envoy\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.066982 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.066918 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.067421 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.067396 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-data\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.067522 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.067435 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.067702 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.067680 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-certs\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.068833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.068806 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istiod-ca-cert\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.069072 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.069049 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-credential-socket\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.070683 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.070656 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-podinfo\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.072243 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.072214 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-envoy\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.089647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.089618 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hpv4\" (UniqueName: \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-kube-api-access-7hpv4\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.091352 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.091329 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-token\") pod \"e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.155123 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.155092 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:02.168313 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.168268 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2118e800-48fe-4900-a656-247cc5fa085a-istio-podinfo\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.168551 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.168379 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-data\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.168551 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.168406 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-istio-token\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.168551 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.168459 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.168551 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.168518 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-envoy\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.168551 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.168548 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-credential-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.169083 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.168588 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2118e800-48fe-4900-a656-247cc5fa085a-istiod-ca-cert\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.169083 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.168676 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-certs\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.169083 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.168746 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4vk\" (UniqueName: \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-kube-api-access-ss4vk\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.270493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.270026 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-data\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.270493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.270076 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-istio-token\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.270493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.270128 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.270493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.270191 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-envoy\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.270493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.270223 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-credential-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.270493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.270261 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2118e800-48fe-4900-a656-247cc5fa085a-istiod-ca-cert\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.270493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.270354 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-certs\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.270493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.270398 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4vk\" (UniqueName: \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-kube-api-access-ss4vk\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.271667 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.271093 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2118e800-48fe-4900-a656-247cc5fa085a-istio-podinfo\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.274618 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.272549 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-credential-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.274618 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.272822 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-certs\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.274618 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.273354 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-data\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.274618 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.273833 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.275154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.275110 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-envoy\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.275649 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.275627 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2118e800-48fe-4900-a656-247cc5fa085a-istiod-ca-cert\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.278310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.278286 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2118e800-48fe-4900-a656-247cc5fa085a-istio-podinfo\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.308529 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.308455 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4vk\" (UniqueName: \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-kube-api-access-ss4vk\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.310927 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.310896 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-istio-token\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.320102 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.319818 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:02.497590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.497548 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb"] Oct 30 14:25:02.498097 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:02.497907 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010bc0ef_5c41_40b4_b6a7_ee2b49eddd90.slice/crio-9d2b162291ad325d345b813b8e12eeff39fac622f450ef3ddd436e7a74db00a0 WatchSource:0}: Error finding container 9d2b162291ad325d345b813b8e12eeff39fac622f450ef3ddd436e7a74db00a0: Status 404 returned error can't find the container with id 9d2b162291ad325d345b813b8e12eeff39fac622f450ef3ddd436e7a74db00a0 Oct 30 14:25:02.500643 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.500610 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:02.500738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.500673 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:02.500738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.500702 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:02.569930 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.569892 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" event={"ID":"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90","Type":"ContainerStarted","Data":"9d2b162291ad325d345b813b8e12eeff39fac622f450ef3ddd436e7a74db00a0"} Oct 30 14:25:02.571473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.571448 4968 generic.go:356] "Generic (PLEG): container finished" podID="afef9040-cff0-40c0-b50a-9f0aea368e8d" containerID="8dba7b8e855cc93aca1521cb66a61128774a50a976f41d15a51c92ebc83ddf4b" exitCode=0 Oct 30 14:25:02.571591 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.571525 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" event={"ID":"afef9040-cff0-40c0-b50a-9f0aea368e8d","Type":"ContainerDied","Data":"8dba7b8e855cc93aca1521cb66a61128774a50a976f41d15a51c92ebc83ddf4b"} Oct 30 14:25:02.572967 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.572948 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" Oct 30 14:25:02.572967 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.572960 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb" event={"ID":"3bbba234-6f5d-46ba-beae-72844a22252e","Type":"ContainerDied","Data":"aa6c78530cd312c9c981b997059b5166c9eca0c90a07d7238ef7be6b613edcd1"} Oct 30 14:25:02.573126 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.573000 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa6c78530cd312c9c981b997059b5166c9eca0c90a07d7238ef7be6b613edcd1" Oct 30 14:25:02.574638 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.574616 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" event={"ID":"57f7ff69-5fde-4505-ae97-bb9ed86bda5d","Type":"ContainerStarted","Data":"287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0"} Oct 30 14:25:02.574762 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.574654 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" event={"ID":"57f7ff69-5fde-4505-ae97-bb9ed86bda5d","Type":"ContainerStarted","Data":"18076ccf34b126c689adba97104f88b95ec77ec5e604e0cd701293143181a176"} Oct 30 14:25:02.669397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.669365 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5"] Oct 30 14:25:02.675743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.675665 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:02.675847 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.675819 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:02.675925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.675864 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:02.707765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.706816 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:02.726106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:02.726054 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" podStartSLOduration=1.726038617 podStartE2EDuration="1.726038617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:02.725504918 +0000 UTC m=+1098.350948291" watchObservedRunningTime="2026-10-30 14:25:02.726038617 +0000 UTC m=+1098.351481989" Oct 30 14:25:03.614742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:03.614635 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" event={"ID":"2118e800-48fe-4900-a656-247cc5fa085a","Type":"ContainerStarted","Data":"a4454be6f86b42b9449b6003a2a589fc6efc1c83ad13cce70b526079c2158bbd"} Oct 30 14:25:03.614742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:03.614706 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" event={"ID":"2118e800-48fe-4900-a656-247cc5fa085a","Type":"ContainerStarted","Data":"2e01c5c40a0ade74d1b986f2ce922296e210bc9b770d857248cbc7f5c2a8e613"} Oct 30 14:25:03.637748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:03.635632 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" event={"ID":"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90","Type":"ContainerStarted","Data":"6411995df9ab33485e1e1dc0bfcee82487ff5a23a26c11ac02ca44bbfd95af72"} Oct 30 14:25:03.728397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:03.728355 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:25:03.728677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:03.728649 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 14:25:03.771385 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:03.771329 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" podStartSLOduration=2.765692265 podStartE2EDuration="2.765692265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:03.728004181 +0000 UTC m=+1099.353447582" watchObservedRunningTime="2026-10-30 14:25:03.765692265 +0000 UTC m=+1099.391135640" Oct 30 14:25:03.834747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:03.834013 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" podStartSLOduration=2.833991203 podStartE2EDuration="2.833991203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:03.811162566 +0000 UTC m=+1099.436605941" watchObservedRunningTime="2026-10-30 14:25:03.833991203 +0000 UTC m=+1099.459434576" Oct 30 14:25:04.191183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.191146 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 14:25:04.218047 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.217991 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 14:25:04.218521 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.191876 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:04.354165 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.337172 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:04.375322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.375289 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 14:25:04.376605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.376575 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 14:25:04.466334 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.465944 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:04.525233 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.525195 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-openshift-ca\") pod \"afef9040-cff0-40c0-b50a-9f0aea368e8d\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " Oct 30 14:25:04.526530 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.526501 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afef9040-cff0-40c0-b50a-9f0aea368e8d-kube-api-access-mqdmq\" (UniqueName: \"kubernetes.io/projected/afef9040-cff0-40c0-b50a-9f0aea368e8d-kube-api-access-mqdmq\") pod \"afef9040-cff0-40c0-b50a-9f0aea368e8d\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " Oct 30 14:25:04.526871 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.526853 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-xks-ca\") pod \"afef9040-cff0-40c0-b50a-9f0aea368e8d\" (UID: \"afef9040-cff0-40c0-b50a-9f0aea368e8d\") " Oct 30 14:25:04.603440 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.603157 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-openshift-ca" pod "afef9040-cff0-40c0-b50a-9f0aea368e8d" (UID: "afef9040-cff0-40c0-b50a-9f0aea368e8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:04.638906 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.638840 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-xks-ca" pod "afef9040-cff0-40c0-b50a-9f0aea368e8d" (UID: "afef9040-cff0-40c0-b50a-9f0aea368e8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:04.648090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.647988 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:04.648283 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.648266 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afef9040-cff0-40c0-b50a-9f0aea368e8d-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:04.751738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.740171 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" Oct 30 14:25:04.818151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.793809 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:25:04.818151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.793877 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 14:25:04.818151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.796871 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" podUID="819c0707-1f66-400e-8704-c902586cae50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:04.818151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.811910 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afef9040-cff0-40c0-b50a-9f0aea368e8d-kube-api-access-mqdmq" pod "afef9040-cff0-40c0-b50a-9f0aea368e8d" (UID: "afef9040-cff0-40c0-b50a-9f0aea368e8d"). InnerVolumeSpecName "kube-api-access-mqdmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:04.819413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.819383 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv" event={"ID":"afef9040-cff0-40c0-b50a-9f0aea368e8d","Type":"ContainerDied","Data":"2a99e0c5828028d9f50fc4952f6a40b7d9f5fbe443e3b981524be811b8983bae"} Oct 30 14:25:04.819621 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.819599 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a99e0c5828028d9f50fc4952f6a40b7d9f5fbe443e3b981524be811b8983bae" Oct 30 14:25:04.894483 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:04.894370 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqdmq\" (UniqueName: \"kubernetes.io/projected/afef9040-cff0-40c0-b50a-9f0aea368e8d-kube-api-access-mqdmq\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:05.068526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:05.068473 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" podUID="554e6fd9-60f6-49bb-a2a4-ef944cc98136" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:05.170912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:05.170817 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 14:25:05.172493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:05.172434 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 14:25:05.376261 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:05.376224 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 14:25:05.430478 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:05.430374 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 14:25:05.706128 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:05.706040 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:25:05.735684 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:05.735643 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 14:25:06.168118 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:06.168079 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 14:25:06.168383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:06.168360 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 14:25:06.432940 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:06.432850 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 14:25:06.442928 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:06.436898 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 14:25:06.726790 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:06.723595 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:25:06.759866 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:06.759816 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 14:25:07.176304 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:07.176087 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 14:25:07.176640 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:07.176593 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 14:25:07.329282 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:07.329248 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 14:25:07.329594 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:07.329542 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 14:25:07.687078 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:07.687039 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:25:07.687370 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:07.687340 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 14:25:08.187147 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.187106 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 14:25:08.209789 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.195236 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 14:25:08.326739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.326275 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 14:25:08.326739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.326338 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 14:25:08.712742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.702115 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:25:08.712742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.702174 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 14:25:08.712742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.712379 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq"] Oct 30 14:25:08.734740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.730915 4968 cpu_manager.go:414] "containerMap: removing container" podUID="afef9040-cff0-40c0-b50a-9f0aea368e8d" containerName="revoke-keys" Oct 30 14:25:08.734740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.730949 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afef9040-cff0-40c0-b50a-9f0aea368e8d" containerName="revoke-keys" Oct 30 14:25:08.734740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.731131 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="afef9040-cff0-40c0-b50a-9f0aea368e8d" containerName="revoke-keys" Oct 30 14:25:08.763756 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.763332 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:08.764912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.764885 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq"] Oct 30 14:25:08.808751 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.786459 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:25:08.808751 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.791149 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:25:08.808751 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.791234 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:25:08.808751 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.791298 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:25:08.954739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.953818 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc95p\" (UniqueName: \"kubernetes.io/projected/99216a44-0d65-4f90-8d7b-5ee32b038d52-kube-api-access-nc95p\") pod \"maas-api-revoke-sub-fdffe94f-cn2bq\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:08.954739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.953996 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdffe94f-cn2bq\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:08.962929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:08.962840 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdffe94f-cn2bq\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:09.065778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.065744 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc95p\" (UniqueName: \"kubernetes.io/projected/99216a44-0d65-4f90-8d7b-5ee32b038d52-kube-api-access-nc95p\") pod \"maas-api-revoke-sub-fdffe94f-cn2bq\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:09.065931 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.065833 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdffe94f-cn2bq\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:09.065931 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.065917 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdffe94f-cn2bq\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:09.066698 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.066668 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdffe94f-cn2bq\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:09.072747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.068531 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdffe94f-cn2bq\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:09.115157 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.111793 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc95p\" (UniqueName: \"kubernetes.io/projected/99216a44-0d65-4f90-8d7b-5ee32b038d52-kube-api-access-nc95p\") pod \"maas-api-revoke-sub-fdffe94f-cn2bq\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:09.156087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.156041 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 14:25:09.156293 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.156108 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 14:25:09.321373 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.320837 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 14:25:09.321373 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.320899 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 14:25:09.411239 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.411202 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:09.687921 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.687307 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:25:09.687921 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.687365 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 14:25:09.801707 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:09.801673 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99216a44_0d65_4f90_8d7b_5ee32b038d52.slice/crio-8b0bcddc7e15e596f37aea7f557debbb67a88802ee72b572ef156c617ad00152 WatchSource:0}: Error finding container 8b0bcddc7e15e596f37aea7f557debbb67a88802ee72b572ef156c617ad00152: Status 404 returned error can't find the container with id 8b0bcddc7e15e596f37aea7f557debbb67a88802ee72b572ef156c617ad00152 Oct 30 14:25:09.811937 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:09.811902 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq"] Oct 30 14:25:10.156263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:10.156229 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 14:25:10.156537 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:10.156492 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 14:25:10.314107 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:10.313909 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" event={"ID":"99216a44-0d65-4f90-8d7b-5ee32b038d52","Type":"ContainerStarted","Data":"bb8052c22c1d9e6baca2473b8d40421f243e199337a14c74eacf0cf8336320db"} Oct 30 14:25:10.314388 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:10.314333 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" event={"ID":"99216a44-0d65-4f90-8d7b-5ee32b038d52","Type":"ContainerStarted","Data":"8b0bcddc7e15e596f37aea7f557debbb67a88802ee72b572ef156c617ad00152"} Oct 30 14:25:10.334113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:10.332413 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 14:25:10.334113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:10.332474 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 14:25:10.733764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:10.731123 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:10.733764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:10.732560 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:10.734959 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:10.733779 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:25:10.792589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:10.792529 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" podStartSLOduration=2.792494546 podStartE2EDuration="2.792494546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:10.368168918 +0000 UTC m=+1105.993612292" watchObservedRunningTime="2026-10-30 14:25:10.792494546 +0000 UTC m=+1106.417937922" Oct 30 14:25:11.160078 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:11.160050 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:11.160511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:11.160486 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:11.161215 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:11.161196 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:25:11.325588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:11.325355 4968 generic.go:356] "Generic (PLEG): container finished" podID="99216a44-0d65-4f90-8d7b-5ee32b038d52" containerID="bb8052c22c1d9e6baca2473b8d40421f243e199337a14c74eacf0cf8336320db" exitCode=0 Oct 30 14:25:11.325588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:11.325576 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" event={"ID":"99216a44-0d65-4f90-8d7b-5ee32b038d52","Type":"ContainerDied","Data":"bb8052c22c1d9e6baca2473b8d40421f243e199337a14c74eacf0cf8336320db"} Oct 30 14:25:11.326035 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:11.326015 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:11.326709 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:11.326687 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:11.327383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:11.327367 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:25:12.499221 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:12.499199 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:12.544541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:12.544499 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-openshift-ca\") pod \"99216a44-0d65-4f90-8d7b-5ee32b038d52\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " Oct 30 14:25:12.544819 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:12.544800 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99216a44-0d65-4f90-8d7b-5ee32b038d52-kube-api-access-nc95p\" (UniqueName: \"kubernetes.io/projected/99216a44-0d65-4f90-8d7b-5ee32b038d52-kube-api-access-nc95p\") pod \"99216a44-0d65-4f90-8d7b-5ee32b038d52\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " Oct 30 14:25:12.544907 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:12.544840 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-xks-ca\") pod \"99216a44-0d65-4f90-8d7b-5ee32b038d52\" (UID: \"99216a44-0d65-4f90-8d7b-5ee32b038d52\") " Oct 30 14:25:12.544907 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:12.544848 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-openshift-ca" pod "99216a44-0d65-4f90-8d7b-5ee32b038d52" (UID: "99216a44-0d65-4f90-8d7b-5ee32b038d52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:12.545707 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:12.545678 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:12.547058 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:12.547029 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-xks-ca" pod "99216a44-0d65-4f90-8d7b-5ee32b038d52" (UID: "99216a44-0d65-4f90-8d7b-5ee32b038d52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:12.547165 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:12.547062 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99216a44-0d65-4f90-8d7b-5ee32b038d52-kube-api-access-nc95p" pod "99216a44-0d65-4f90-8d7b-5ee32b038d52" (UID: "99216a44-0d65-4f90-8d7b-5ee32b038d52"). InnerVolumeSpecName "kube-api-access-nc95p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:12.647286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:12.647207 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc95p\" (UniqueName: \"kubernetes.io/projected/99216a44-0d65-4f90-8d7b-5ee32b038d52-kube-api-access-nc95p\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:12.647286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:12.647236 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99216a44-0d65-4f90-8d7b-5ee32b038d52-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:13.340931 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.340898 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" Oct 30 14:25:13.341086 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.340929 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq" event={"ID":"99216a44-0d65-4f90-8d7b-5ee32b038d52","Type":"ContainerDied","Data":"8b0bcddc7e15e596f37aea7f557debbb67a88802ee72b572ef156c617ad00152"} Oct 30 14:25:13.341086 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.341016 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b0bcddc7e15e596f37aea7f557debbb67a88802ee72b572ef156c617ad00152" Oct 30 14:25:13.549396 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.549363 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng"] Oct 30 14:25:13.549958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.549945 4968 cpu_manager.go:414] "containerMap: removing container" podUID="99216a44-0d65-4f90-8d7b-5ee32b038d52" containerName="revoke-keys" Oct 30 14:25:13.550015 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.549959 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99216a44-0d65-4f90-8d7b-5ee32b038d52" containerName="revoke-keys" Oct 30 14:25:13.550071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.550062 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="99216a44-0d65-4f90-8d7b-5ee32b038d52" containerName="revoke-keys" Oct 30 14:25:13.553765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.553746 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:13.569197 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.569173 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:25:13.569359 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.569218 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:25:13.574748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.574710 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng"] Oct 30 14:25:13.588754 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.579090 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:25:13.588754 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.579157 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:25:13.659144 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.659065 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eefc604d-sk9ng\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:13.659144 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.659105 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdks\" (UniqueName: \"kubernetes.io/projected/d6d762a3-661e-4b35-a6b4-14b60f966066-kube-api-access-gzdks\") pod \"maas-api-revoke-sub-eefc604d-sk9ng\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:13.659144 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.659134 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eefc604d-sk9ng\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:13.760355 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.760316 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eefc604d-sk9ng\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:13.760633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.760606 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdks\" (UniqueName: \"kubernetes.io/projected/d6d762a3-661e-4b35-a6b4-14b60f966066-kube-api-access-gzdks\") pod \"maas-api-revoke-sub-eefc604d-sk9ng\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:13.760791 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.760670 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eefc604d-sk9ng\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:13.762189 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.762164 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eefc604d-sk9ng\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:13.764073 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.764051 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eefc604d-sk9ng\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:13.782983 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.782956 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdks\" (UniqueName: \"kubernetes.io/projected/d6d762a3-661e-4b35-a6b4-14b60f966066-kube-api-access-gzdks\") pod \"maas-api-revoke-sub-eefc604d-sk9ng\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:13.869845 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:13.869811 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:14.178670 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:14.178603 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng"] Oct 30 14:25:14.180444 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:14.180416 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d762a3_661e_4b35_a6b4_14b60f966066.slice/crio-28b22268db480eaaf847febe77b9121110cc126a5c6e002f93fdea9ee67e9050 WatchSource:0}: Error finding container 28b22268db480eaaf847febe77b9121110cc126a5c6e002f93fdea9ee67e9050: Status 404 returned error can't find the container with id 28b22268db480eaaf847febe77b9121110cc126a5c6e002f93fdea9ee67e9050 Oct 30 14:25:14.351595 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:14.351562 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" event={"ID":"d6d762a3-661e-4b35-a6b4-14b60f966066","Type":"ContainerStarted","Data":"28b22268db480eaaf847febe77b9121110cc126a5c6e002f93fdea9ee67e9050"} Oct 30 14:25:15.147415 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.147378 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg"] Oct 30 14:25:15.152737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.152700 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:15.173690 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.173662 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg"] Oct 30 14:25:15.177177 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.177149 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2de7231-eb29-4092-beec-43b325415a52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:15.177322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.177296 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d5lk\" (UniqueName: \"kubernetes.io/projected/b2de7231-eb29-4092-beec-43b325415a52-kube-api-access-4d5lk\") pod \"maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:15.177463 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.177445 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2de7231-eb29-4092-beec-43b325415a52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:15.278364 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.278272 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2de7231-eb29-4092-beec-43b325415a52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:15.278524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.278408 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2de7231-eb29-4092-beec-43b325415a52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:15.278524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.278507 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d5lk\" (UniqueName: \"kubernetes.io/projected/b2de7231-eb29-4092-beec-43b325415a52-kube-api-access-4d5lk\") pod \"maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:15.279124 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.279096 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2de7231-eb29-4092-beec-43b325415a52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:15.281196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.281171 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2de7231-eb29-4092-beec-43b325415a52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:15.308397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.308368 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d5lk\" (UniqueName: \"kubernetes.io/projected/b2de7231-eb29-4092-beec-43b325415a52-kube-api-access-4d5lk\") pod \"maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:15.360259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.360227 4968 generic.go:356] "Generic (PLEG): container finished" podID="d6d762a3-661e-4b35-a6b4-14b60f966066" containerID="ec9242e4a5b07ce7db8bc622508448d8817644548a0699f2c495f6068be89904" exitCode=0 Oct 30 14:25:15.360392 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.360309 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" event={"ID":"d6d762a3-661e-4b35-a6b4-14b60f966066","Type":"ContainerDied","Data":"ec9242e4a5b07ce7db8bc622508448d8817644548a0699f2c495f6068be89904"} Oct 30 14:25:15.467087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.467057 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:15.790640 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:15.790559 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg"] Oct 30 14:25:15.794892 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:15.794851 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2de7231_eb29_4092_beec_43b325415a52.slice/crio-5e391772c5b5ed4c2bfc45ce3bfe7d197c07359e51abc0b2b085d24cbb8dfed6 WatchSource:0}: Error finding container 5e391772c5b5ed4c2bfc45ce3bfe7d197c07359e51abc0b2b085d24cbb8dfed6: Status 404 returned error can't find the container with id 5e391772c5b5ed4c2bfc45ce3bfe7d197c07359e51abc0b2b085d24cbb8dfed6 Oct 30 14:25:16.370032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.369942 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" event={"ID":"b2de7231-eb29-4092-beec-43b325415a52","Type":"ContainerStarted","Data":"d5434bc562471e976455f800d082ab02af6975daacadf017d77b8fbc11a4dfb7"} Oct 30 14:25:16.370032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.369997 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" event={"ID":"b2de7231-eb29-4092-beec-43b325415a52","Type":"ContainerStarted","Data":"5e391772c5b5ed4c2bfc45ce3bfe7d197c07359e51abc0b2b085d24cbb8dfed6"} Oct 30 14:25:16.404632 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.404596 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb"] Oct 30 14:25:16.420642 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.420607 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.425019 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.424962 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" podStartSLOduration=1.424941631 podStartE2EDuration="1.424941631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:16.419526266 +0000 UTC m=+1112.044969640" watchObservedRunningTime="2026-10-30 14:25:16.424941631 +0000 UTC m=+1112.050385008" Oct 30 14:25:16.435367 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.435335 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb"] Oct 30 14:25:16.514652 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.514613 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istiod-ca-cert\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.515103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.515073 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-credential-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.515248 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.515233 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twj4b\" (UniqueName: \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-kube-api-access-twj4b\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.515317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.515293 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-token\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.515385 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.515369 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-podinfo\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.515450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.515408 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-envoy\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.515508 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.515458 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-certs\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.515508 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.515486 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.515608 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.515521 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-data\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.579688 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.579661 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:16.616707 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.616673 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-openshift-ca\") pod \"d6d762a3-661e-4b35-a6b4-14b60f966066\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " Oct 30 14:25:16.616887 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.616818 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d762a3-661e-4b35-a6b4-14b60f966066-kube-api-access-gzdks\" (UniqueName: \"kubernetes.io/projected/d6d762a3-661e-4b35-a6b4-14b60f966066-kube-api-access-gzdks\") pod \"d6d762a3-661e-4b35-a6b4-14b60f966066\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " Oct 30 14:25:16.616961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.616906 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-xks-ca\") pod \"d6d762a3-661e-4b35-a6b4-14b60f966066\" (UID: \"d6d762a3-661e-4b35-a6b4-14b60f966066\") " Oct 30 14:25:16.617032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.617002 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-openshift-ca" pod "d6d762a3-661e-4b35-a6b4-14b60f966066" (UID: "d6d762a3-661e-4b35-a6b4-14b60f966066"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:16.617303 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.617275 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twj4b\" (UniqueName: \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-kube-api-access-twj4b\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.617437 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.617369 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-token\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.617501 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.617457 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-podinfo\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.617501 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.617494 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-envoy\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.617604 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.617549 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-certs\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.617604 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.617588 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.617702 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.617631 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-data\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.617772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.617706 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istiod-ca-cert\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.617772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.617763 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-credential-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.617956 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.617934 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:16.618284 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.618263 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-credential-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.618520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.618501 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-certs\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.618993 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.618736 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-socket\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.618993 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.618979 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-data\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.619672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.619646 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istiod-ca-cert\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.622360 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.621332 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-envoy\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.622360 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.621828 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d762a3-661e-4b35-a6b4-14b60f966066-kube-api-access-gzdks" pod "d6d762a3-661e-4b35-a6b4-14b60f966066" (UID: "d6d762a3-661e-4b35-a6b4-14b60f966066"). InnerVolumeSpecName "kube-api-access-gzdks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:16.622360 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.622095 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-podinfo\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.622360 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.622203 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-xks-ca" pod "d6d762a3-661e-4b35-a6b4-14b60f966066" (UID: "d6d762a3-661e-4b35-a6b4-14b60f966066"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:16.641117 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.638563 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twj4b\" (UniqueName: \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-kube-api-access-twj4b\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.641117 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.640375 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-token\") pod \"e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:16.719627 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.719587 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzdks\" (UniqueName: \"kubernetes.io/projected/d6d762a3-661e-4b35-a6b4-14b60f966066-kube-api-access-gzdks\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:16.719627 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.719620 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d762a3-661e-4b35-a6b4-14b60f966066-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:16.749393 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:16.749362 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:17.085202 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.085172 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb"] Oct 30 14:25:17.087168 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:17.087137 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432f1cdf_16a2_4e08_aa85_9b8c7ab0c4ad.slice/crio-cb5c489aaab6553529669c4b5de0fec8215e4276116d6803bce68adbbd6175e5 WatchSource:0}: Error finding container cb5c489aaab6553529669c4b5de0fec8215e4276116d6803bce68adbbd6175e5: Status 404 returned error can't find the container with id cb5c489aaab6553529669c4b5de0fec8215e4276116d6803bce68adbbd6175e5 Oct 30 14:25:17.090060 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.090013 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:17.090139 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.090118 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:17.090185 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.090158 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:25:17.390811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.381419 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" Oct 30 14:25:17.390811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.381415 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng" event={"ID":"d6d762a3-661e-4b35-a6b4-14b60f966066","Type":"ContainerDied","Data":"28b22268db480eaaf847febe77b9121110cc126a5c6e002f93fdea9ee67e9050"} Oct 30 14:25:17.390811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.381574 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28b22268db480eaaf847febe77b9121110cc126a5c6e002f93fdea9ee67e9050" Oct 30 14:25:17.390811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.383380 4968 generic.go:356] "Generic (PLEG): container finished" podID="b2de7231-eb29-4092-beec-43b325415a52" containerID="d5434bc562471e976455f800d082ab02af6975daacadf017d77b8fbc11a4dfb7" exitCode=0 Oct 30 14:25:17.390811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.383468 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" event={"ID":"b2de7231-eb29-4092-beec-43b325415a52","Type":"ContainerDied","Data":"d5434bc562471e976455f800d082ab02af6975daacadf017d77b8fbc11a4dfb7"} Oct 30 14:25:17.390811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.386224 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" event={"ID":"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad","Type":"ContainerStarted","Data":"f5ec0fc9a536c00189405058d8cff95084f3d57d6e9b55c6d876ab4345b3808a"} Oct 30 14:25:17.390811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.386268 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" event={"ID":"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad","Type":"ContainerStarted","Data":"cb5c489aaab6553529669c4b5de0fec8215e4276116d6803bce68adbbd6175e5"} Oct 30 14:25:17.478350 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.478295 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" podStartSLOduration=1.478280196 podStartE2EDuration="1.478280196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:17.473767834 +0000 UTC m=+1113.099211206" watchObservedRunningTime="2026-10-30 14:25:17.478280196 +0000 UTC m=+1113.103723621" Oct 30 14:25:17.751983 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.751884 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:17.816867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.816825 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h"] Oct 30 14:25:17.817453 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.817437 4968 cpu_manager.go:414] "containerMap: removing container" podUID="d6d762a3-661e-4b35-a6b4-14b60f966066" containerName="revoke-keys" Oct 30 14:25:17.817453 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.817453 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d762a3-661e-4b35-a6b4-14b60f966066" containerName="revoke-keys" Oct 30 14:25:17.817621 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.817561 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d762a3-661e-4b35-a6b4-14b60f966066" containerName="revoke-keys" Oct 30 14:25:17.821542 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.821520 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:17.832966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.832942 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h"] Oct 30 14:25:17.938843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.938807 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxhfq\" (UniqueName: \"kubernetes.io/projected/27fd7342-ae8c-4515-af80-7df58976e7e6-kube-api-access-cxhfq\") pod \"maas-api-revoke-sub-bde66139-2x72h\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:17.938843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.938846 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bde66139-2x72h\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:17.939064 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:17.939006 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bde66139-2x72h\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:18.040584 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.040495 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bde66139-2x72h\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:18.040753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.040640 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bde66139-2x72h\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:18.040875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.040859 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxhfq\" (UniqueName: \"kubernetes.io/projected/27fd7342-ae8c-4515-af80-7df58976e7e6-kube-api-access-cxhfq\") pod \"maas-api-revoke-sub-bde66139-2x72h\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:18.041306 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.041280 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bde66139-2x72h\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:18.043139 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.043109 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bde66139-2x72h\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:18.062835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.062813 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxhfq\" (UniqueName: \"kubernetes.io/projected/27fd7342-ae8c-4515-af80-7df58976e7e6-kube-api-access-cxhfq\") pod \"maas-api-revoke-sub-bde66139-2x72h\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:18.139722 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.139677 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:18.212841 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.212803 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:25:18.499660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.495929 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h"] Oct 30 14:25:18.632733 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.632695 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:18.750530 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.750494 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2de7231-eb29-4092-beec-43b325415a52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2de7231-eb29-4092-beec-43b325415a52-maas-api-openshift-ca\") pod \"b2de7231-eb29-4092-beec-43b325415a52\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " Oct 30 14:25:18.751242 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.751197 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2de7231-eb29-4092-beec-43b325415a52-maas-api-openshift-ca" pod "b2de7231-eb29-4092-beec-43b325415a52" (UID: "b2de7231-eb29-4092-beec-43b325415a52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:18.751492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.751219 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2de7231-eb29-4092-beec-43b325415a52-kube-api-access-4d5lk\" (UniqueName: \"kubernetes.io/projected/b2de7231-eb29-4092-beec-43b325415a52-kube-api-access-4d5lk\") pod \"b2de7231-eb29-4092-beec-43b325415a52\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " Oct 30 14:25:18.751779 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.751568 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 14:25:18.751910 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.751833 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2de7231-eb29-4092-beec-43b325415a52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2de7231-eb29-4092-beec-43b325415a52-maas-api-xks-ca\") pod \"b2de7231-eb29-4092-beec-43b325415a52\" (UID: \"b2de7231-eb29-4092-beec-43b325415a52\") " Oct 30 14:25:18.752362 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.751828 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 14:25:18.752510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.752484 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2de7231-eb29-4092-beec-43b325415a52-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:18.754084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.754060 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2de7231-eb29-4092-beec-43b325415a52-maas-api-xks-ca" pod "b2de7231-eb29-4092-beec-43b325415a52" (UID: "b2de7231-eb29-4092-beec-43b325415a52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:18.754194 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.754173 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2de7231-eb29-4092-beec-43b325415a52-kube-api-access-4d5lk" pod "b2de7231-eb29-4092-beec-43b325415a52" (UID: "b2de7231-eb29-4092-beec-43b325415a52"). InnerVolumeSpecName "kube-api-access-4d5lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:18.854004 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.853971 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d5lk\" (UniqueName: \"kubernetes.io/projected/b2de7231-eb29-4092-beec-43b325415a52-kube-api-access-4d5lk\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:18.854004 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:18.853997 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2de7231-eb29-4092-beec-43b325415a52-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:19.481566 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:19.481270 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" event={"ID":"b2de7231-eb29-4092-beec-43b325415a52","Type":"ContainerDied","Data":"5e391772c5b5ed4c2bfc45ce3bfe7d197c07359e51abc0b2b085d24cbb8dfed6"} Oct 30 14:25:19.481566 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:19.481336 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e391772c5b5ed4c2bfc45ce3bfe7d197c07359e51abc0b2b085d24cbb8dfed6" Oct 30 14:25:19.481566 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:19.481412 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg" Oct 30 14:25:19.537487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:19.536968 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" event={"ID":"27fd7342-ae8c-4515-af80-7df58976e7e6","Type":"ContainerStarted","Data":"0c121ad35f675fd7403e14db1fa590b770ad008de6db1435a1a99e428e49efc0"} Oct 30 14:25:19.537487 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:19.537047 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" event={"ID":"27fd7342-ae8c-4515-af80-7df58976e7e6","Type":"ContainerStarted","Data":"1adfd8d9216009bb18d7b1cdbb3d425ce072504a76e1affc5b807bc68acd97ea"} Oct 30 14:25:19.579050 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:19.578988 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" podStartSLOduration=2.578969113 podStartE2EDuration="2.578969113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:19.573791148 +0000 UTC m=+1115.199234532" watchObservedRunningTime="2026-10-30 14:25:19.578969113 +0000 UTC m=+1115.204412487" Oct 30 14:25:19.750209 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:19.750091 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 14:25:19.750209 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:19.750170 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 14:25:19.942738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:19.941912 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4"] Oct 30 14:25:19.948691 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:19.948401 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q"] Oct 30 14:25:20.663044 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:20.663004 4968 generic.go:356] "Generic (PLEG): container finished" podID="27fd7342-ae8c-4515-af80-7df58976e7e6" containerID="0c121ad35f675fd7403e14db1fa590b770ad008de6db1435a1a99e428e49efc0" exitCode=0 Oct 30 14:25:20.663629 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:20.663603 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" event={"ID":"27fd7342-ae8c-4515-af80-7df58976e7e6","Type":"ContainerDied","Data":"0c121ad35f675fd7403e14db1fa590b770ad008de6db1435a1a99e428e49efc0"} Oct 30 14:25:20.667351 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:25:20.667322 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:25:20.765155 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:20.765116 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 14:25:20.772873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:20.772826 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 14:25:21.680891 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:21.680821 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" podUID="48066b06-babd-49b6-961a-1bf34b8c7973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:21.837752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:21.834842 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 14:25:21.837752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:21.834927 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 14:25:22.236947 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.236913 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:22.272855 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.270393 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-openshift-ca\") pod \"27fd7342-ae8c-4515-af80-7df58976e7e6\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " Oct 30 14:25:22.272855 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.270537 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-xks-ca\") pod \"27fd7342-ae8c-4515-af80-7df58976e7e6\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " Oct 30 14:25:22.272855 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.270675 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27fd7342-ae8c-4515-af80-7df58976e7e6-kube-api-access-cxhfq\" (UniqueName: \"kubernetes.io/projected/27fd7342-ae8c-4515-af80-7df58976e7e6-kube-api-access-cxhfq\") pod \"27fd7342-ae8c-4515-af80-7df58976e7e6\" (UID: \"27fd7342-ae8c-4515-af80-7df58976e7e6\") " Oct 30 14:25:22.347391 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.347129 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-openshift-ca" pod "27fd7342-ae8c-4515-af80-7df58976e7e6" (UID: "27fd7342-ae8c-4515-af80-7df58976e7e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:22.352333 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.352306 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fd7342-ae8c-4515-af80-7df58976e7e6-kube-api-access-cxhfq" pod "27fd7342-ae8c-4515-af80-7df58976e7e6" (UID: "27fd7342-ae8c-4515-af80-7df58976e7e6"). InnerVolumeSpecName "kube-api-access-cxhfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:22.356360 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.356312 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-xks-ca" pod "27fd7342-ae8c-4515-af80-7df58976e7e6" (UID: "27fd7342-ae8c-4515-af80-7df58976e7e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:22.374768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.374734 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:22.374914 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.374783 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fd7342-ae8c-4515-af80-7df58976e7e6-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:22.374914 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.374800 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxhfq\" (UniqueName: \"kubernetes.io/projected/27fd7342-ae8c-4515-af80-7df58976e7e6-kube-api-access-cxhfq\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:22.739904 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.739862 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" event={"ID":"27fd7342-ae8c-4515-af80-7df58976e7e6","Type":"ContainerDied","Data":"1adfd8d9216009bb18d7b1cdbb3d425ce072504a76e1affc5b807bc68acd97ea"} Oct 30 14:25:22.740309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.739929 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1adfd8d9216009bb18d7b1cdbb3d425ce072504a76e1affc5b807bc68acd97ea" Oct 30 14:25:22.740309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.739936 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h" Oct 30 14:25:22.750193 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.750167 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 14:25:22.750324 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:22.750209 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 14:25:23.750513 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:23.750482 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 14:25:23.751016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:23.750534 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 14:25:24.075223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.075131 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4"] Oct 30 14:25:24.075865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.075843 4968 cpu_manager.go:414] "containerMap: removing container" podUID="27fd7342-ae8c-4515-af80-7df58976e7e6" containerName="revoke-keys" Oct 30 14:25:24.075865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.075864 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fd7342-ae8c-4515-af80-7df58976e7e6" containerName="revoke-keys" Oct 30 14:25:24.076033 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.075886 4968 cpu_manager.go:414] "containerMap: removing container" podUID="b2de7231-eb29-4092-beec-43b325415a52" containerName="revoke-keys" Oct 30 14:25:24.076033 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.075893 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2de7231-eb29-4092-beec-43b325415a52" containerName="revoke-keys" Oct 30 14:25:24.076129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.076068 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fd7342-ae8c-4515-af80-7df58976e7e6" containerName="revoke-keys" Oct 30 14:25:24.076129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.076082 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2de7231-eb29-4092-beec-43b325415a52" containerName="revoke-keys" Oct 30 14:25:24.080017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.079990 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:24.105645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.104578 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:25:24.105645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.104769 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:25:24.105645 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.104876 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:25:24.133241 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.108604 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:25:24.173780 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.165762 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4"] Oct 30 14:25:24.229603 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.229562 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb268ef-s65k4\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:24.230005 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.229985 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrpxc\" (UniqueName: \"kubernetes.io/projected/5060dc40-67c7-4d61-8829-a84041f026e1-kube-api-access-lrpxc\") pod \"maas-api-revoke-sub-5fb268ef-s65k4\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:24.230213 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.230197 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb268ef-s65k4\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:24.331834 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.331792 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpxc\" (UniqueName: \"kubernetes.io/projected/5060dc40-67c7-4d61-8829-a84041f026e1-kube-api-access-lrpxc\") pod \"maas-api-revoke-sub-5fb268ef-s65k4\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:24.331995 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.331908 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb268ef-s65k4\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:24.332072 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.332056 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb268ef-s65k4\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:24.332762 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.332703 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb268ef-s65k4\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:24.334494 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.334470 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb268ef-s65k4\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:24.356601 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.356572 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrpxc\" (UniqueName: \"kubernetes.io/projected/5060dc40-67c7-4d61-8829-a84041f026e1-kube-api-access-lrpxc\") pod \"maas-api-revoke-sub-5fb268ef-s65k4\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:24.398942 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.398910 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:24.623995 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.623911 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx"] Oct 30 14:25:24.629317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.629293 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:24.635225 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.635192 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzghg\" (UniqueName: \"kubernetes.io/projected/231675b8-2cda-4d8f-ab74-03164765f8d3-kube-api-access-mzghg\") pod \"maas-api-revoke-sub-72adc37f-w2wfx\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:24.635378 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.635351 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72adc37f-w2wfx\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:24.635476 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.635431 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72adc37f-w2wfx\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:24.650370 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.650345 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx"] Oct 30 14:25:24.715314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.715286 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4"] Oct 30 14:25:24.718159 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:24.718110 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5060dc40_67c7_4d61_8829_a84041f026e1.slice/crio-c5bafe17aafd92379d63f3866edf5f2c18f805b292d2830d10230d27977caf8b WatchSource:0}: Error finding container c5bafe17aafd92379d63f3866edf5f2c18f805b292d2830d10230d27977caf8b: Status 404 returned error can't find the container with id c5bafe17aafd92379d63f3866edf5f2c18f805b292d2830d10230d27977caf8b Oct 30 14:25:24.736442 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.736411 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzghg\" (UniqueName: \"kubernetes.io/projected/231675b8-2cda-4d8f-ab74-03164765f8d3-kube-api-access-mzghg\") pod \"maas-api-revoke-sub-72adc37f-w2wfx\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:24.736573 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.736543 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72adc37f-w2wfx\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:24.736635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.736604 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72adc37f-w2wfx\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:24.737445 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.737418 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72adc37f-w2wfx\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:24.739045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.739024 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72adc37f-w2wfx\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:24.750059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.750035 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 14:25:24.750171 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.750083 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 14:25:24.756993 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.756964 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" event={"ID":"5060dc40-67c7-4d61-8829-a84041f026e1","Type":"ContainerStarted","Data":"c5bafe17aafd92379d63f3866edf5f2c18f805b292d2830d10230d27977caf8b"} Oct 30 14:25:24.758511 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.758492 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzghg\" (UniqueName: \"kubernetes.io/projected/231675b8-2cda-4d8f-ab74-03164765f8d3-kube-api-access-mzghg\") pod \"maas-api-revoke-sub-72adc37f-w2wfx\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:24.945049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:24.945018 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:25.270377 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:25.270340 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx"] Oct 30 14:25:25.272871 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:25.272846 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231675b8_2cda_4d8f_ab74_03164765f8d3.slice/crio-2e32aa3405e348c27fadb0d0eb8e552c431bf50a8ecf49bdda0c074166cbc54a WatchSource:0}: Error finding container 2e32aa3405e348c27fadb0d0eb8e552c431bf50a8ecf49bdda0c074166cbc54a: Status 404 returned error can't find the container with id 2e32aa3405e348c27fadb0d0eb8e552c431bf50a8ecf49bdda0c074166cbc54a Oct 30 14:25:25.671667 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:25.671628 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb"] Oct 30 14:25:25.768268 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:25.768223 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" event={"ID":"5060dc40-67c7-4d61-8829-a84041f026e1","Type":"ContainerStarted","Data":"8e0f86e8fb8214489906e27ae81687b25ab86d1d0177a93b04f825cca7a8b547"} Oct 30 14:25:25.770232 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:25.770208 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" event={"ID":"231675b8-2cda-4d8f-ab74-03164765f8d3","Type":"ContainerStarted","Data":"8b0ab3e410d90b68937ebeee35b2e09d927b6c859960c8ff053d08b5e2e45d38"} Oct 30 14:25:25.770397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:25.770386 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" event={"ID":"231675b8-2cda-4d8f-ab74-03164765f8d3","Type":"ContainerStarted","Data":"2e32aa3405e348c27fadb0d0eb8e552c431bf50a8ecf49bdda0c074166cbc54a"} Oct 30 14:25:25.885806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:25.883356 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" podStartSLOduration=1.883339533 podStartE2EDuration="1.883339533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:25.83877385 +0000 UTC m=+1121.464217224" watchObservedRunningTime="2026-10-30 14:25:25.883339533 +0000 UTC m=+1121.508782907" Oct 30 14:25:25.885806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:25.885677 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" podStartSLOduration=1.885660842 podStartE2EDuration="1.885660842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:25.882107384 +0000 UTC m=+1121.507550762" watchObservedRunningTime="2026-10-30 14:25:25.885660842 +0000 UTC m=+1121.511104216" Oct 30 14:25:26.780151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:26.780107 4968 generic.go:356] "Generic (PLEG): container finished" podID="231675b8-2cda-4d8f-ab74-03164765f8d3" containerID="8b0ab3e410d90b68937ebeee35b2e09d927b6c859960c8ff053d08b5e2e45d38" exitCode=0 Oct 30 14:25:26.780611 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:26.780389 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" event={"ID":"231675b8-2cda-4d8f-ab74-03164765f8d3","Type":"ContainerDied","Data":"8b0ab3e410d90b68937ebeee35b2e09d927b6c859960c8ff053d08b5e2e45d38"} Oct 30 14:25:26.782589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:26.782555 4968 generic.go:356] "Generic (PLEG): container finished" podID="5060dc40-67c7-4d61-8829-a84041f026e1" containerID="8e0f86e8fb8214489906e27ae81687b25ab86d1d0177a93b04f825cca7a8b547" exitCode=0 Oct 30 14:25:26.782703 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:26.782663 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" event={"ID":"5060dc40-67c7-4d61-8829-a84041f026e1","Type":"ContainerDied","Data":"8e0f86e8fb8214489906e27ae81687b25ab86d1d0177a93b04f825cca7a8b547"} Oct 30 14:25:26.793988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:26.793872 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:25:26.795245 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:26.794157 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:27.045755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:27.045643 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:27.046159 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:27.046145 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:27.047009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:27.046986 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:25:28.082807 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.082775 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:28.086104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.086085 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:28.117919 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.117892 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5060dc40-67c7-4d61-8829-a84041f026e1-kube-api-access-lrpxc\" (UniqueName: \"kubernetes.io/projected/5060dc40-67c7-4d61-8829-a84041f026e1-kube-api-access-lrpxc\") pod \"5060dc40-67c7-4d61-8829-a84041f026e1\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " Oct 30 14:25:28.118064 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.117937 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-xks-ca\") pod \"5060dc40-67c7-4d61-8829-a84041f026e1\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " Oct 30 14:25:28.118121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.118064 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-openshift-ca\") pod \"231675b8-2cda-4d8f-ab74-03164765f8d3\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " Oct 30 14:25:28.118164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.118131 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-xks-ca\") pod \"231675b8-2cda-4d8f-ab74-03164765f8d3\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " Oct 30 14:25:28.118215 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.118188 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-openshift-ca\") pod \"5060dc40-67c7-4d61-8829-a84041f026e1\" (UID: \"5060dc40-67c7-4d61-8829-a84041f026e1\") " Oct 30 14:25:28.118253 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.118221 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231675b8-2cda-4d8f-ab74-03164765f8d3-kube-api-access-mzghg\" (UniqueName: \"kubernetes.io/projected/231675b8-2cda-4d8f-ab74-03164765f8d3-kube-api-access-mzghg\") pod \"231675b8-2cda-4d8f-ab74-03164765f8d3\" (UID: \"231675b8-2cda-4d8f-ab74-03164765f8d3\") " Oct 30 14:25:28.118522 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.118496 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-openshift-ca" pod "231675b8-2cda-4d8f-ab74-03164765f8d3" (UID: "231675b8-2cda-4d8f-ab74-03164765f8d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:28.118647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.118588 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-openshift-ca" pod "5060dc40-67c7-4d61-8829-a84041f026e1" (UID: "5060dc40-67c7-4d61-8829-a84041f026e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:28.118981 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.118957 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:28.119090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.118986 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:28.120162 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.120124 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-xks-ca" pod "5060dc40-67c7-4d61-8829-a84041f026e1" (UID: "5060dc40-67c7-4d61-8829-a84041f026e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:28.120567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.120480 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231675b8-2cda-4d8f-ab74-03164765f8d3-kube-api-access-mzghg" pod "231675b8-2cda-4d8f-ab74-03164765f8d3" (UID: "231675b8-2cda-4d8f-ab74-03164765f8d3"). InnerVolumeSpecName "kube-api-access-mzghg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:28.120567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.120547 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5060dc40-67c7-4d61-8829-a84041f026e1-kube-api-access-lrpxc" pod "5060dc40-67c7-4d61-8829-a84041f026e1" (UID: "5060dc40-67c7-4d61-8829-a84041f026e1"). InnerVolumeSpecName "kube-api-access-lrpxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:28.120670 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.120561 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-xks-ca" pod "231675b8-2cda-4d8f-ab74-03164765f8d3" (UID: "231675b8-2cda-4d8f-ab74-03164765f8d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:28.220210 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.220178 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrpxc\" (UniqueName: \"kubernetes.io/projected/5060dc40-67c7-4d61-8829-a84041f026e1-kube-api-access-lrpxc\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:28.220210 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.220206 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5060dc40-67c7-4d61-8829-a84041f026e1-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:28.220210 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.220215 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231675b8-2cda-4d8f-ab74-03164765f8d3-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:28.220429 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.220225 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzghg\" (UniqueName: \"kubernetes.io/projected/231675b8-2cda-4d8f-ab74-03164765f8d3-kube-api-access-mzghg\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:28.812374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.812286 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" Oct 30 14:25:28.812374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.812314 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4" event={"ID":"5060dc40-67c7-4d61-8829-a84041f026e1","Type":"ContainerDied","Data":"c5bafe17aafd92379d63f3866edf5f2c18f805b292d2830d10230d27977caf8b"} Oct 30 14:25:28.812598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.812390 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5bafe17aafd92379d63f3866edf5f2c18f805b292d2830d10230d27977caf8b" Oct 30 14:25:28.814272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.814247 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" Oct 30 14:25:28.814451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.814286 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx" event={"ID":"231675b8-2cda-4d8f-ab74-03164765f8d3","Type":"ContainerDied","Data":"2e32aa3405e348c27fadb0d0eb8e552c431bf50a8ecf49bdda0c074166cbc54a"} Oct 30 14:25:28.814451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:28.814326 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e32aa3405e348c27fadb0d0eb8e552c431bf50a8ecf49bdda0c074166cbc54a" Oct 30 14:25:29.182397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.182354 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j"] Oct 30 14:25:29.183116 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.183097 4968 cpu_manager.go:414] "containerMap: removing container" podUID="231675b8-2cda-4d8f-ab74-03164765f8d3" containerName="revoke-keys" Oct 30 14:25:29.183168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.183118 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231675b8-2cda-4d8f-ab74-03164765f8d3" containerName="revoke-keys" Oct 30 14:25:29.183168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.183156 4968 cpu_manager.go:414] "containerMap: removing container" podUID="5060dc40-67c7-4d61-8829-a84041f026e1" containerName="revoke-keys" Oct 30 14:25:29.183168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.183162 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5060dc40-67c7-4d61-8829-a84041f026e1" containerName="revoke-keys" Oct 30 14:25:29.183265 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.183253 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="5060dc40-67c7-4d61-8829-a84041f026e1" containerName="revoke-keys" Oct 30 14:25:29.183265 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.183261 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="231675b8-2cda-4d8f-ab74-03164765f8d3" containerName="revoke-keys" Oct 30 14:25:29.186575 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.186560 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:29.196360 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.196341 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk"] Oct 30 14:25:29.201424 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.201394 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:29.201560 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.201541 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:25:29.205516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.205460 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:25:29.205638 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.205542 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:25:29.205806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.205782 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:25:29.207489 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.207464 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j"] Oct 30 14:25:29.223757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.222588 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk"] Oct 30 14:25:29.233006 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.232980 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktpq\" (UniqueName: \"kubernetes.io/projected/2b442a1b-7616-4e5b-ac26-e5e60723e882-kube-api-access-sktpq\") pod \"maas-api-revoke-sub-9bc44f81-bhx6j\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:29.233112 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.233030 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c624f054-b42dk\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:29.233112 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.233101 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c624f054-b42dk\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:29.233192 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.233120 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bc44f81-bhx6j\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:29.233235 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.233223 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzvl\" (UniqueName: \"kubernetes.io/projected/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-kube-api-access-dmzvl\") pod \"maas-api-revoke-sub-c624f054-b42dk\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:29.233277 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.233267 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bc44f81-bhx6j\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:29.334742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.334682 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktpq\" (UniqueName: \"kubernetes.io/projected/2b442a1b-7616-4e5b-ac26-e5e60723e882-kube-api-access-sktpq\") pod \"maas-api-revoke-sub-9bc44f81-bhx6j\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:29.334921 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.334752 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c624f054-b42dk\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:29.334921 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.334779 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c624f054-b42dk\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:29.334921 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.334798 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bc44f81-bhx6j\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:29.334921 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.334891 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzvl\" (UniqueName: \"kubernetes.io/projected/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-kube-api-access-dmzvl\") pod \"maas-api-revoke-sub-c624f054-b42dk\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:29.335150 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.334941 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bc44f81-bhx6j\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:29.335540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.335506 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c624f054-b42dk\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:29.335790 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.335757 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bc44f81-bhx6j\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:29.337513 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.337482 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bc44f81-bhx6j\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:29.337614 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.337600 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c624f054-b42dk\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:29.357943 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.357916 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktpq\" (UniqueName: \"kubernetes.io/projected/2b442a1b-7616-4e5b-ac26-e5e60723e882-kube-api-access-sktpq\") pod \"maas-api-revoke-sub-9bc44f81-bhx6j\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:29.358032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.357976 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzvl\" (UniqueName: \"kubernetes.io/projected/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-kube-api-access-dmzvl\") pod \"maas-api-revoke-sub-c624f054-b42dk\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:29.503559 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.503478 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:29.525811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.525788 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:29.867191 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:29.867159 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b442a1b_7616_4e5b_ac26_e5e60723e882.slice/crio-a7ccc3734eeda63dc8684cf62fbde4b37150e1f429b87b29314fd08926c7a778 WatchSource:0}: Error finding container a7ccc3734eeda63dc8684cf62fbde4b37150e1f429b87b29314fd08926c7a778: Status 404 returned error can't find the container with id a7ccc3734eeda63dc8684cf62fbde4b37150e1f429b87b29314fd08926c7a778 Oct 30 14:25:29.867852 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.867733 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j"] Oct 30 14:25:29.948103 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:29.948077 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96e3e13_1d90_4ab9_8539_70730cb8a3a3.slice/crio-1c5bb17d0af19d61b785e9f6402c49e08ec5403dd8ab7356632c7ed3805f526e WatchSource:0}: Error finding container 1c5bb17d0af19d61b785e9f6402c49e08ec5403dd8ab7356632c7ed3805f526e: Status 404 returned error can't find the container with id 1c5bb17d0af19d61b785e9f6402c49e08ec5403dd8ab7356632c7ed3805f526e Oct 30 14:25:29.950314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:29.950286 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk"] Oct 30 14:25:30.672833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:30.672769 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" podUID="3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2" containerName="payload-processing" containerID="cri-o://287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a" gracePeriod=25 Oct 30 14:25:30.842532 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:30.842496 4968 generic.go:356] "Generic (PLEG): container finished" podID="2b442a1b-7616-4e5b-ac26-e5e60723e882" containerID="6da50b2cd572072418f95ad6caf89e591f2f1a3ab6d5ffa7508934e8a06a73c5" exitCode=0 Oct 30 14:25:30.842673 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:30.842567 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" event={"ID":"2b442a1b-7616-4e5b-ac26-e5e60723e882","Type":"ContainerDied","Data":"6da50b2cd572072418f95ad6caf89e591f2f1a3ab6d5ffa7508934e8a06a73c5"} Oct 30 14:25:30.842673 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:30.842645 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" event={"ID":"2b442a1b-7616-4e5b-ac26-e5e60723e882","Type":"ContainerStarted","Data":"a7ccc3734eeda63dc8684cf62fbde4b37150e1f429b87b29314fd08926c7a778"} Oct 30 14:25:30.844366 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:30.844344 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" event={"ID":"f96e3e13-1d90-4ab9-8539-70730cb8a3a3","Type":"ContainerStarted","Data":"df54c2d9c97256d825e609485084482e3ebaf0f7a6393097c38f713ff2909fcc"} Oct 30 14:25:30.844473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:30.844392 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" event={"ID":"f96e3e13-1d90-4ab9-8539-70730cb8a3a3","Type":"ContainerStarted","Data":"1c5bb17d0af19d61b785e9f6402c49e08ec5403dd8ab7356632c7ed3805f526e"} Oct 30 14:25:30.919297 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:30.919235 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" podStartSLOduration=1.919220736 podStartE2EDuration="1.919220736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:30.917004943 +0000 UTC m=+1126.542448315" watchObservedRunningTime="2026-10-30 14:25:30.919220736 +0000 UTC m=+1126.544664106" Oct 30 14:25:31.853998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:31.853959 4968 generic.go:356] "Generic (PLEG): container finished" podID="f96e3e13-1d90-4ab9-8539-70730cb8a3a3" containerID="df54c2d9c97256d825e609485084482e3ebaf0f7a6393097c38f713ff2909fcc" exitCode=0 Oct 30 14:25:31.854408 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:31.854042 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" event={"ID":"f96e3e13-1d90-4ab9-8539-70730cb8a3a3","Type":"ContainerDied","Data":"df54c2d9c97256d825e609485084482e3ebaf0f7a6393097c38f713ff2909fcc"} Oct 30 14:25:32.040869 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.040843 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:32.084850 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.084819 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-xks-ca\") pod \"2b442a1b-7616-4e5b-ac26-e5e60723e882\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " Oct 30 14:25:32.087012 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.086981 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-openshift-ca\") pod \"2b442a1b-7616-4e5b-ac26-e5e60723e882\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " Oct 30 14:25:32.087169 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.087150 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b442a1b-7616-4e5b-ac26-e5e60723e882-kube-api-access-sktpq\" (UniqueName: \"kubernetes.io/projected/2b442a1b-7616-4e5b-ac26-e5e60723e882-kube-api-access-sktpq\") pod \"2b442a1b-7616-4e5b-ac26-e5e60723e882\" (UID: \"2b442a1b-7616-4e5b-ac26-e5e60723e882\") " Oct 30 14:25:32.097311 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.088353 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-openshift-ca" pod "2b442a1b-7616-4e5b-ac26-e5e60723e882" (UID: "2b442a1b-7616-4e5b-ac26-e5e60723e882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:32.097644 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.097620 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-xks-ca" pod "2b442a1b-7616-4e5b-ac26-e5e60723e882" (UID: "2b442a1b-7616-4e5b-ac26-e5e60723e882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:32.098067 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.098048 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b442a1b-7616-4e5b-ac26-e5e60723e882-kube-api-access-sktpq" pod "2b442a1b-7616-4e5b-ac26-e5e60723e882" (UID: "2b442a1b-7616-4e5b-ac26-e5e60723e882"). InnerVolumeSpecName "kube-api-access-sktpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:32.189351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.189309 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:32.189351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.189348 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sktpq\" (UniqueName: \"kubernetes.io/projected/2b442a1b-7616-4e5b-ac26-e5e60723e882-kube-api-access-sktpq\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:32.189506 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.189363 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b442a1b-7616-4e5b-ac26-e5e60723e882-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:32.866289 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.866261 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" Oct 30 14:25:32.866695 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.866297 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j" event={"ID":"2b442a1b-7616-4e5b-ac26-e5e60723e882","Type":"ContainerDied","Data":"a7ccc3734eeda63dc8684cf62fbde4b37150e1f429b87b29314fd08926c7a778"} Oct 30 14:25:32.866695 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:32.866344 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7ccc3734eeda63dc8684cf62fbde4b37150e1f429b87b29314fd08926c7a778" Oct 30 14:25:33.143762 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.143736 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:33.221868 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.221833 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-openshift-ca\") pod \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " Oct 30 14:25:33.222048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.221891 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-xks-ca\") pod \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " Oct 30 14:25:33.222144 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.222123 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-kube-api-access-dmzvl\" (UniqueName: \"kubernetes.io/projected/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-kube-api-access-dmzvl\") pod \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\" (UID: \"f96e3e13-1d90-4ab9-8539-70730cb8a3a3\") " Oct 30 14:25:33.222251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.222231 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-openshift-ca" pod "f96e3e13-1d90-4ab9-8539-70730cb8a3a3" (UID: "f96e3e13-1d90-4ab9-8539-70730cb8a3a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:33.222660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.222633 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:33.224848 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.224822 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-kube-api-access-dmzvl" pod "f96e3e13-1d90-4ab9-8539-70730cb8a3a3" (UID: "f96e3e13-1d90-4ab9-8539-70730cb8a3a3"). InnerVolumeSpecName "kube-api-access-dmzvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:33.225087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.225050 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-xks-ca" pod "f96e3e13-1d90-4ab9-8539-70730cb8a3a3" (UID: "f96e3e13-1d90-4ab9-8539-70730cb8a3a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:33.323994 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.323965 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:33.323994 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.323991 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmzvl\" (UniqueName: \"kubernetes.io/projected/f96e3e13-1d90-4ab9-8539-70730cb8a3a3-kube-api-access-dmzvl\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:33.876438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.876409 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" Oct 30 14:25:33.876895 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.876441 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk" event={"ID":"f96e3e13-1d90-4ab9-8539-70730cb8a3a3","Type":"ContainerDied","Data":"1c5bb17d0af19d61b785e9f6402c49e08ec5403dd8ab7356632c7ed3805f526e"} Oct 30 14:25:33.876895 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:33.876488 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c5bb17d0af19d61b785e9f6402c49e08ec5403dd8ab7356632c7ed3805f526e" Oct 30 14:25:34.217685 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:34.217591 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" podUID="3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:25:34.974878 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:34.974843 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" podUID="9f8a5dab-f785-409d-bb57-0459127c0656" containerName="main" containerID="cri-o://5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919" gracePeriod=45 Oct 30 14:25:34.979156 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:34.979121 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" podUID="99b00b7d-0f8b-41f7-b646-995797d34045" containerName="main" containerID="cri-o://531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75" gracePeriod=45 Oct 30 14:25:35.050954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.050922 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" podUID="99b00b7d-0f8b-41f7-b646-995797d34045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": dial tcp 10.132.0.63:8000: connect: connection refused" Oct 30 14:25:35.293754 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.293702 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:25:35.304121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.304095 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:25:35.323584 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.323542 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p"] Oct 30 14:25:35.324229 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324207 4968 cpu_manager.go:414] "containerMap: removing container" podUID="f96e3e13-1d90-4ab9-8539-70730cb8a3a3" containerName="revoke-keys" Oct 30 14:25:35.324332 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324230 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96e3e13-1d90-4ab9-8539-70730cb8a3a3" containerName="revoke-keys" Oct 30 14:25:35.324332 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324277 4968 cpu_manager.go:414] "containerMap: removing container" podUID="2b442a1b-7616-4e5b-ac26-e5e60723e882" containerName="revoke-keys" Oct 30 14:25:35.324332 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324286 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b442a1b-7616-4e5b-ac26-e5e60723e882" containerName="revoke-keys" Oct 30 14:25:35.324332 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324328 4968 cpu_manager.go:414] "containerMap: removing container" podUID="9f8a5dab-f785-409d-bb57-0459127c0656" containerName="main" Oct 30 14:25:35.324574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324335 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f8a5dab-f785-409d-bb57-0459127c0656" containerName="main" Oct 30 14:25:35.324574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324351 4968 cpu_manager.go:414] "containerMap: removing container" podUID="99b00b7d-0f8b-41f7-b646-995797d34045" containerName="main" Oct 30 14:25:35.324574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324359 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b00b7d-0f8b-41f7-b646-995797d34045" containerName="main" Oct 30 14:25:35.324574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324510 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b442a1b-7616-4e5b-ac26-e5e60723e882" containerName="revoke-keys" Oct 30 14:25:35.324574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324532 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f8a5dab-f785-409d-bb57-0459127c0656" containerName="main" Oct 30 14:25:35.324574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324544 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b00b7d-0f8b-41f7-b646-995797d34045" containerName="main" Oct 30 14:25:35.324574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.324555 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96e3e13-1d90-4ab9-8539-70730cb8a3a3" containerName="revoke-keys" Oct 30 14:25:35.328169 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.328147 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:35.345875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.345839 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:25:35.346146 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.346123 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:25:35.346847 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.346188 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:25:35.346847 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.346217 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:25:35.349636 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.349610 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p"] Oct 30 14:25:35.458426 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.458388 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-model-cache\") pod \"9f8a5dab-f785-409d-bb57-0459127c0656\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " Oct 30 14:25:35.458589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.458503 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-dshm\") pod \"9f8a5dab-f785-409d-bb57-0459127c0656\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " Oct 30 14:25:35.458589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.458538 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-model-cache\") pod \"99b00b7d-0f8b-41f7-b646-995797d34045\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " Oct 30 14:25:35.458759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.458604 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-tmp-dir\") pod \"99b00b7d-0f8b-41f7-b646-995797d34045\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " Oct 30 14:25:35.458759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.458637 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f8a5dab-f785-409d-bb57-0459127c0656-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8a5dab-f785-409d-bb57-0459127c0656-tls-certs\") pod \"9f8a5dab-f785-409d-bb57-0459127c0656\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " Oct 30 14:25:35.458759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.458690 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-model-cache" pod "9f8a5dab-f785-409d-bb57-0459127c0656" (UID: "9f8a5dab-f785-409d-bb57-0459127c0656"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:35.458759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.458702 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99b00b7d-0f8b-41f7-b646-995797d34045-tls-certs\" (UniqueName: \"kubernetes.io/secret/99b00b7d-0f8b-41f7-b646-995797d34045-tls-certs\") pod \"99b00b7d-0f8b-41f7-b646-995797d34045\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " Oct 30 14:25:35.458991 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.458842 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b00b7d-0f8b-41f7-b646-995797d34045-kube-api-access-xwkl7\" (UniqueName: \"kubernetes.io/projected/99b00b7d-0f8b-41f7-b646-995797d34045-kube-api-access-xwkl7\") pod \"99b00b7d-0f8b-41f7-b646-995797d34045\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " Oct 30 14:25:35.458991 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.458886 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-home\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-home\") pod \"99b00b7d-0f8b-41f7-b646-995797d34045\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " Oct 30 14:25:35.458991 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.458972 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-tmp-dir\") pod \"9f8a5dab-f785-409d-bb57-0459127c0656\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " Oct 30 14:25:35.459153 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.459066 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-home\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-home\") pod \"9f8a5dab-f785-409d-bb57-0459127c0656\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " Oct 30 14:25:35.459153 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.459108 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8a5dab-f785-409d-bb57-0459127c0656-kube-api-access-8g55p\" (UniqueName: \"kubernetes.io/projected/9f8a5dab-f785-409d-bb57-0459127c0656-kube-api-access-8g55p\") pod \"9f8a5dab-f785-409d-bb57-0459127c0656\" (UID: \"9f8a5dab-f785-409d-bb57-0459127c0656\") " Oct 30 14:25:35.459153 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.459140 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-dshm\") pod \"99b00b7d-0f8b-41f7-b646-995797d34045\" (UID: \"99b00b7d-0f8b-41f7-b646-995797d34045\") " Oct 30 14:25:35.459604 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.459578 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:35.459799 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.459778 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:35.460007 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.459104 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-tmp-dir" pod "99b00b7d-0f8b-41f7-b646-995797d34045" (UID: "99b00b7d-0f8b-41f7-b646-995797d34045"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:35.460112 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.459266 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-model-cache" pod "99b00b7d-0f8b-41f7-b646-995797d34045" (UID: "99b00b7d-0f8b-41f7-b646-995797d34045"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:35.460195 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.459433 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-tmp-dir" pod "9f8a5dab-f785-409d-bb57-0459127c0656" (UID: "9f8a5dab-f785-409d-bb57-0459127c0656"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:35.460291 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.459668 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-home" pod "9f8a5dab-f785-409d-bb57-0459127c0656" (UID: "9f8a5dab-f785-409d-bb57-0459127c0656"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:35.460436 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.459971 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-home" pod "99b00b7d-0f8b-41f7-b646-995797d34045" (UID: "99b00b7d-0f8b-41f7-b646-995797d34045"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:35.460553 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.460117 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7q5f\" (UniqueName: \"kubernetes.io/projected/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-kube-api-access-l7q5f\") pod \"maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:35.460868 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.460848 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.460982 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.460969 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.461080 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.461070 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.461244 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.461231 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.461372 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.461360 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.461541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.461527 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.462582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.462543 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-dshm" pod "99b00b7d-0f8b-41f7-b646-995797d34045" (UID: "99b00b7d-0f8b-41f7-b646-995797d34045"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:35.462906 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.462872 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8a5dab-f785-409d-bb57-0459127c0656-kube-api-access-8g55p" pod "9f8a5dab-f785-409d-bb57-0459127c0656" (UID: "9f8a5dab-f785-409d-bb57-0459127c0656"). InnerVolumeSpecName "kube-api-access-8g55p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:35.463023 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.462985 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-dshm" pod "9f8a5dab-f785-409d-bb57-0459127c0656" (UID: "9f8a5dab-f785-409d-bb57-0459127c0656"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:35.463087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.463007 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b00b7d-0f8b-41f7-b646-995797d34045-tls-certs" pod "99b00b7d-0f8b-41f7-b646-995797d34045" (UID: "99b00b7d-0f8b-41f7-b646-995797d34045"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:35.464567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.464524 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b00b7d-0f8b-41f7-b646-995797d34045-kube-api-access-xwkl7" pod "99b00b7d-0f8b-41f7-b646-995797d34045" (UID: "99b00b7d-0f8b-41f7-b646-995797d34045"). InnerVolumeSpecName "kube-api-access-xwkl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:35.464763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.464706 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f8a5dab-f785-409d-bb57-0459127c0656-tls-certs" pod "9f8a5dab-f785-409d-bb57-0459127c0656" (UID: "9f8a5dab-f785-409d-bb57-0459127c0656"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:35.563258 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.563165 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:35.563258 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.563205 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:35.563455 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.563288 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7q5f\" (UniqueName: \"kubernetes.io/projected/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-kube-api-access-l7q5f\") pod \"maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:35.563455 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.563418 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g55p\" (UniqueName: \"kubernetes.io/projected/9f8a5dab-f785-409d-bb57-0459127c0656-kube-api-access-8g55p\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.563455 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.563435 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b00b7d-0f8b-41f7-b646-995797d34045-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.563455 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.563454 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8a5dab-f785-409d-bb57-0459127c0656-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.563580 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.563470 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8a5dab-f785-409d-bb57-0459127c0656-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.563580 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.563483 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b00b7d-0f8b-41f7-b646-995797d34045-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.563580 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.563495 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwkl7\" (UniqueName: \"kubernetes.io/projected/99b00b7d-0f8b-41f7-b646-995797d34045-kube-api-access-xwkl7\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:35.563904 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.563866 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:35.565825 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.565805 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:35.599047 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.599017 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7q5f\" (UniqueName: \"kubernetes.io/projected/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-kube-api-access-l7q5f\") pod \"maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:35.642626 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.642597 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:35.894782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.894742 4968 generic.go:356] "Generic (PLEG): container finished" podID="9f8a5dab-f785-409d-bb57-0459127c0656" containerID="5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919" exitCode=0 Oct 30 14:25:35.895025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.895004 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" event={"ID":"9f8a5dab-f785-409d-bb57-0459127c0656","Type":"ContainerDied","Data":"5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919"} Oct 30 14:25:35.895132 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.895050 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" event={"ID":"9f8a5dab-f785-409d-bb57-0459127c0656","Type":"ContainerDied","Data":"cc92ff1e620fee705d68646e54dccfb29306a94964bd46619c67c1961e32ca9b"} Oct 30 14:25:35.895132 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.895075 4968 scope.go:122] "RemoveContainer" containerID="5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919" Oct 30 14:25:35.895242 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.895228 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4" Oct 30 14:25:35.901495 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.901464 4968 generic.go:356] "Generic (PLEG): container finished" podID="99b00b7d-0f8b-41f7-b646-995797d34045" containerID="531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75" exitCode=0 Oct 30 14:25:35.901696 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.901549 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" Oct 30 14:25:35.901696 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.901585 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" event={"ID":"99b00b7d-0f8b-41f7-b646-995797d34045","Type":"ContainerDied","Data":"531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75"} Oct 30 14:25:35.901696 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.901617 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q" event={"ID":"99b00b7d-0f8b-41f7-b646-995797d34045","Type":"ContainerDied","Data":"44cfd24fcc9749faa4ee3b148c0d85877fe0c8187f32850391e057252330c584"} Oct 30 14:25:35.919161 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.917747 4968 scope.go:122] "RemoveContainer" containerID="5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919" Oct 30 14:25:35.919161 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:25:35.918110 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919\": container with ID starting with 5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919 not found: ID does not exist" containerID="5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919" Oct 30 14:25:35.919161 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.918146 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919"} err="failed to get container status \"5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919\": rpc error: code = NotFound desc = could not find container \"5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919\": container with ID starting with 5c0f7bd490eacd0dd987d64355c3d6aaff612a6b1afd5762c93d3c598689d919 not found: ID does not exist" Oct 30 14:25:35.919161 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.918191 4968 scope.go:122] "RemoveContainer" containerID="531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75" Oct 30 14:25:35.932984 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.932965 4968 scope.go:122] "RemoveContainer" containerID="531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75" Oct 30 14:25:35.933298 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:25:35.933268 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75\": container with ID starting with 531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75 not found: ID does not exist" containerID="531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75" Oct 30 14:25:35.933412 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.933305 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75"} err="failed to get container status \"531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75\": rpc error: code = NotFound desc = could not find container \"531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75\": container with ID starting with 531f4d59447ccafb2ef6d727c4594da2aa24594ccff525db2a9bdac84165cc75 not found: ID does not exist" Oct 30 14:25:35.963696 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.963663 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4"] Oct 30 14:25:35.968544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.968522 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/distinct-w2-a5ddf3-kserve-5d4cf4557d-26rp4"] Oct 30 14:25:35.973057 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:35.973033 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p"] Oct 30 14:25:35.978150 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:35.978128 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bc4c5d_0a42_4842_aba5_4ba1d15aeb2c.slice/crio-30f20a87f1fad4247e6525e84df61f764013b4b03caa599b32df78917ba969e6 WatchSource:0}: Error finding container 30f20a87f1fad4247e6525e84df61f764013b4b03caa599b32df78917ba969e6: Status 404 returned error can't find the container with id 30f20a87f1fad4247e6525e84df61f764013b4b03caa599b32df78917ba969e6 Oct 30 14:25:36.001038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:36.000928 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q"] Oct 30 14:25:36.025417 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:36.025381 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a5ddf3/premium-w2-a5ddf3-kserve-f5765d85f-j4x4q"] Oct 30 14:25:36.521324 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:36.521231 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b00b7d-0f8b-41f7-b646-995797d34045" path="/var/lib/kubelet/pods/99b00b7d-0f8b-41f7-b646-995797d34045/volumes" Oct 30 14:25:36.521740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:36.521710 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f8a5dab-f785-409d-bb57-0459127c0656" path="/var/lib/kubelet/pods/9f8a5dab-f785-409d-bb57-0459127c0656/volumes" Oct 30 14:25:36.913616 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:36.913578 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" event={"ID":"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c","Type":"ContainerStarted","Data":"219d2978550e97ff6c5a79639cc1dc544717889d85710e8a1c98d183efff983b"} Oct 30 14:25:36.913813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:36.913670 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" event={"ID":"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c","Type":"ContainerStarted","Data":"30f20a87f1fad4247e6525e84df61f764013b4b03caa599b32df78917ba969e6"} Oct 30 14:25:36.952049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:36.951982 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" podStartSLOduration=1.951964137 podStartE2EDuration="1.951964137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:36.949866993 +0000 UTC m=+1132.575310365" watchObservedRunningTime="2026-10-30 14:25:36.951964137 +0000 UTC m=+1132.577407510" Oct 30 14:25:37.924994 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:37.924963 4968 generic.go:356] "Generic (PLEG): container finished" podID="38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c" containerID="219d2978550e97ff6c5a79639cc1dc544717889d85710e8a1c98d183efff983b" exitCode=0 Oct 30 14:25:37.925397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:37.925047 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" event={"ID":"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c","Type":"ContainerDied","Data":"219d2978550e97ff6c5a79639cc1dc544717889d85710e8a1c98d183efff983b"} Oct 30 14:25:39.098620 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.098592 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:39.220380 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.220287 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-xks-ca\") pod \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " Oct 30 14:25:39.221221 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.220380 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-kube-api-access-l7q5f\" (UniqueName: \"kubernetes.io/projected/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-kube-api-access-l7q5f\") pod \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " Oct 30 14:25:39.221221 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.220539 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-openshift-ca\") pod \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\" (UID: \"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c\") " Oct 30 14:25:39.221221 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.221010 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-openshift-ca" pod "38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c" (UID: "38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:39.221221 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.221116 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:39.222671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.222648 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-xks-ca" pod "38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c" (UID: "38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:39.223151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.223127 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-kube-api-access-l7q5f" pod "38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c" (UID: "38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c"). InnerVolumeSpecName "kube-api-access-l7q5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:39.234693 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.234669 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp"] Oct 30 14:25:39.235630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.235611 4968 cpu_manager.go:414] "containerMap: removing container" podUID="38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c" containerName="revoke-keys" Oct 30 14:25:39.235676 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.235635 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c" containerName="revoke-keys" Oct 30 14:25:39.235838 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.235824 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c" containerName="revoke-keys" Oct 30 14:25:39.239644 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.239628 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:39.257028 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.257003 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp"] Oct 30 14:25:39.322104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.322071 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:39.322104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.322098 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7q5f\" (UniqueName: \"kubernetes.io/projected/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c-kube-api-access-l7q5f\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:39.423208 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.423177 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s2gb\" (UniqueName: \"kubernetes.io/projected/9dd8c616-3730-4f1e-b0e5-614c7f407612-kube-api-access-8s2gb\") pod \"maas-api-revoke-sub-399a47bf-b2fdp\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:39.423344 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.423307 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-399a47bf-b2fdp\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:39.423396 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.423376 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-399a47bf-b2fdp\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:39.525160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.525080 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s2gb\" (UniqueName: \"kubernetes.io/projected/9dd8c616-3730-4f1e-b0e5-614c7f407612-kube-api-access-8s2gb\") pod \"maas-api-revoke-sub-399a47bf-b2fdp\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:39.525298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.525185 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-399a47bf-b2fdp\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:39.525298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.525211 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-399a47bf-b2fdp\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:39.525923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.525900 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-399a47bf-b2fdp\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:39.527496 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.527474 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-399a47bf-b2fdp\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:39.548970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.548948 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s2gb\" (UniqueName: \"kubernetes.io/projected/9dd8c616-3730-4f1e-b0e5-614c7f407612-kube-api-access-8s2gb\") pod \"maas-api-revoke-sub-399a47bf-b2fdp\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:39.569858 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.568249 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:39.941583 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.941547 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp"] Oct 30 14:25:39.992871 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.992825 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" Oct 30 14:25:39.993006 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.992825 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p" event={"ID":"38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c","Type":"ContainerDied","Data":"30f20a87f1fad4247e6525e84df61f764013b4b03caa599b32df78917ba969e6"} Oct 30 14:25:39.993006 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:39.992957 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30f20a87f1fad4247e6525e84df61f764013b4b03caa599b32df78917ba969e6" Oct 30 14:25:40.105352 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:40.104596 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6"] Oct 30 14:25:40.118398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:40.118372 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl"] Oct 30 14:25:40.182745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:40.172791 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6"] Oct 30 14:25:41.075747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.062361 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" event={"ID":"9dd8c616-3730-4f1e-b0e5-614c7f407612","Type":"ContainerStarted","Data":"996cdb93718435f9fd3c72a95148eca08c4c8a51625e605a3537864a5c88d432"} Oct 30 14:25:41.075747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.062433 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" event={"ID":"9dd8c616-3730-4f1e-b0e5-614c7f407612","Type":"ContainerStarted","Data":"5abb190941ff52f23216a19808cab148ef62833550bb23f574560da3e7fbdf93"} Oct 30 14:25:41.105735 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.104623 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" podStartSLOduration=2.104602764 podStartE2EDuration="2.104602764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:41.103763522 +0000 UTC m=+1136.729206900" watchObservedRunningTime="2026-10-30 14:25:41.104602764 +0000 UTC m=+1136.730046137" Oct 30 14:25:41.480741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.479832 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64"] Oct 30 14:25:41.507114 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.507085 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:41.627752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.540976 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64"] Oct 30 14:25:41.627752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.604583 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:41.627752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.604830 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mg58\" (UniqueName: \"kubernetes.io/projected/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-kube-api-access-4mg58\") pod \"maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:41.627752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.604914 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:41.722751 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.715679 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mg58\" (UniqueName: \"kubernetes.io/projected/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-kube-api-access-4mg58\") pod \"maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:41.817454 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.817366 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:41.818513 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.818487 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:41.845966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.818353 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:41.898615 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.898543 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mg58\" (UniqueName: \"kubernetes.io/projected/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-kube-api-access-4mg58\") pod \"maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:41.957740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.953287 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:41.959208 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:41.958027 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" podUID="aca3c382-c905-4be5-ae1a-d5f9336adc1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:42.143372 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:42.143333 4968 generic.go:356] "Generic (PLEG): container finished" podID="9dd8c616-3730-4f1e-b0e5-614c7f407612" containerID="996cdb93718435f9fd3c72a95148eca08c4c8a51625e605a3537864a5c88d432" exitCode=0 Oct 30 14:25:42.144005 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:42.143978 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" event={"ID":"9dd8c616-3730-4f1e-b0e5-614c7f407612","Type":"ContainerDied","Data":"996cdb93718435f9fd3c72a95148eca08c4c8a51625e605a3537864a5c88d432"} Oct 30 14:25:42.218141 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:42.218104 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:25:42.218372 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:42.218346 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:42.241317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:42.241287 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:42.401017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:42.400905 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" podUID="58305994-dcf4-44a4-afeb-b3e89cad85f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:42.837798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:42.802851 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" podUID="182fc12f-335a-4b5c-b26c-6ae65caa58cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:42.837798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:42.803231 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" podUID="819c0707-1f66-400e-8704-c902586cae50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:43.361760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.314194 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" podUID="c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:43.361760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.321610 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" podUID="554e6fd9-60f6-49bb-a2a4-ef944cc98136" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:43.362569 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.362540 4968 patch_prober.go:29] interesting pod/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:25:43.362769 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.362743 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:43.456008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.455957 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" podUID="58305994-dcf4-44a4-afeb-b3e89cad85f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:43.468020 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.467974 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:25:43.822432 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.822387 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" podUID="819c0707-1f66-400e-8704-c902586cae50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:43.913765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.827808 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" podUID="182fc12f-335a-4b5c-b26c-6ae65caa58cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 14:25:43.913765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.828084 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:25:43.913765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.828142 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:25:43.979218 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.979183 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:25:43.996770 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:43.993286 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64"] Oct 30 14:25:44.316332 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.300699 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" podUID="3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:25:44.319934 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.319895 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" podUID="c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:44.324617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.320706 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:25:44.336934 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.336889 4968 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" podUID="c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:25:44.351324 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.351299 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:25:44.353836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.353815 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:25:44.367589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.367558 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:25:44.464880 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.464848 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" event={"ID":"1b1b5298-d140-48d3-830c-ac7c2ec4ef14","Type":"ContainerStarted","Data":"9908dd93887f35b80ed612c661897c112dd94ac3d24fbac02dbd646030942de3"} Oct 30 14:25:44.519031 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.519004 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:44.614499 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.611657 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-openshift-ca\") pod \"9dd8c616-3730-4f1e-b0e5-614c7f407612\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " Oct 30 14:25:44.614499 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.611803 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd8c616-3730-4f1e-b0e5-614c7f407612-kube-api-access-8s2gb\" (UniqueName: \"kubernetes.io/projected/9dd8c616-3730-4f1e-b0e5-614c7f407612-kube-api-access-8s2gb\") pod \"9dd8c616-3730-4f1e-b0e5-614c7f407612\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " Oct 30 14:25:44.614499 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.611973 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-xks-ca\") pod \"9dd8c616-3730-4f1e-b0e5-614c7f407612\" (UID: \"9dd8c616-3730-4f1e-b0e5-614c7f407612\") " Oct 30 14:25:44.614499 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.614035 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-openshift-ca" pod "9dd8c616-3730-4f1e-b0e5-614c7f407612" (UID: "9dd8c616-3730-4f1e-b0e5-614c7f407612"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:44.620416 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.620369 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-xks-ca" pod "9dd8c616-3730-4f1e-b0e5-614c7f407612" (UID: "9dd8c616-3730-4f1e-b0e5-614c7f407612"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:44.620416 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.620374 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd8c616-3730-4f1e-b0e5-614c7f407612-kube-api-access-8s2gb" pod "9dd8c616-3730-4f1e-b0e5-614c7f407612" (UID: "9dd8c616-3730-4f1e-b0e5-614c7f407612"). InnerVolumeSpecName "kube-api-access-8s2gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:44.714230 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.714200 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:44.714343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.714234 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s2gb\" (UniqueName: \"kubernetes.io/projected/9dd8c616-3730-4f1e-b0e5-614c7f407612-kube-api-access-8s2gb\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:44.714343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:44.714250 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd8c616-3730-4f1e-b0e5-614c7f407612-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:45.474079 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:45.474049 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" Oct 30 14:25:45.474558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:45.474074 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp" event={"ID":"9dd8c616-3730-4f1e-b0e5-614c7f407612","Type":"ContainerDied","Data":"5abb190941ff52f23216a19808cab148ef62833550bb23f574560da3e7fbdf93"} Oct 30 14:25:45.474558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:45.474149 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5abb190941ff52f23216a19808cab148ef62833550bb23f574560da3e7fbdf93" Oct 30 14:25:45.475847 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:45.475818 4968 generic.go:356] "Generic (PLEG): container finished" podID="1b1b5298-d140-48d3-830c-ac7c2ec4ef14" containerID="5a3d59e8e44923b181cf41fa600ba6e8b8ebb2019fc0747e27a5c494c0eb46b7" exitCode=0 Oct 30 14:25:45.475992 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:45.475899 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" event={"ID":"1b1b5298-d140-48d3-830c-ac7c2ec4ef14","Type":"ContainerDied","Data":"5a3d59e8e44923b181cf41fa600ba6e8b8ebb2019fc0747e27a5c494c0eb46b7"} Oct 30 14:25:45.843694 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:45.841516 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw"] Oct 30 14:25:46.883970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:46.883935 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:46.966385 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:46.966351 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-openshift-ca\") pod \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " Oct 30 14:25:46.966581 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:46.966398 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-xks-ca\") pod \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " Oct 30 14:25:46.966658 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:46.966625 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-kube-api-access-4mg58\" (UniqueName: \"kubernetes.io/projected/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-kube-api-access-4mg58\") pod \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\" (UID: \"1b1b5298-d140-48d3-830c-ac7c2ec4ef14\") " Oct 30 14:25:46.966835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:46.966801 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-openshift-ca" pod "1b1b5298-d140-48d3-830c-ac7c2ec4ef14" (UID: "1b1b5298-d140-48d3-830c-ac7c2ec4ef14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:46.967229 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:46.967203 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:46.968662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:46.968628 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-xks-ca" pod "1b1b5298-d140-48d3-830c-ac7c2ec4ef14" (UID: "1b1b5298-d140-48d3-830c-ac7c2ec4ef14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:46.969369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:46.969337 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-kube-api-access-4mg58" pod "1b1b5298-d140-48d3-830c-ac7c2ec4ef14" (UID: "1b1b5298-d140-48d3-830c-ac7c2ec4ef14"). InnerVolumeSpecName "kube-api-access-4mg58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:47.068157 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:47.068074 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mg58\" (UniqueName: \"kubernetes.io/projected/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-kube-api-access-4mg58\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:47.068157 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:47.068108 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1b5298-d140-48d3-830c-ac7c2ec4ef14-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:47.494586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:47.494555 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" Oct 30 14:25:47.494788 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:47.494581 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64" event={"ID":"1b1b5298-d140-48d3-830c-ac7c2ec4ef14","Type":"ContainerDied","Data":"9908dd93887f35b80ed612c661897c112dd94ac3d24fbac02dbd646030942de3"} Oct 30 14:25:47.494788 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:47.494658 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9908dd93887f35b80ed612c661897c112dd94ac3d24fbac02dbd646030942de3" Oct 30 14:25:49.771524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:49.771454 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg"] Oct 30 14:25:49.780813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:49.780784 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a5ddf3-tqvwg"] Oct 30 14:25:50.444672 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:25:50.444638 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:25:50.520170 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.520137 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2de7231-eb29-4092-beec-43b325415a52" path="/var/lib/kubelet/pods/b2de7231-eb29-4092-beec-43b325415a52/volumes" Oct 30 14:25:50.592621 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.592581 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv"] Oct 30 14:25:50.593272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.593251 4968 cpu_manager.go:414] "containerMap: removing container" podUID="9dd8c616-3730-4f1e-b0e5-614c7f407612" containerName="revoke-keys" Oct 30 14:25:50.593386 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.593274 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd8c616-3730-4f1e-b0e5-614c7f407612" containerName="revoke-keys" Oct 30 14:25:50.593386 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.593293 4968 cpu_manager.go:414] "containerMap: removing container" podUID="1b1b5298-d140-48d3-830c-ac7c2ec4ef14" containerName="revoke-keys" Oct 30 14:25:50.593386 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.593300 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1b5298-d140-48d3-830c-ac7c2ec4ef14" containerName="revoke-keys" Oct 30 14:25:50.593541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.593505 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd8c616-3730-4f1e-b0e5-614c7f407612" containerName="revoke-keys" Oct 30 14:25:50.593541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.593521 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1b5298-d140-48d3-830c-ac7c2ec4ef14" containerName="revoke-keys" Oct 30 14:25:50.597479 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.597458 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:50.626459 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.624226 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:25:50.626459 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.624246 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:25:50.630619 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.630587 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv"] Oct 30 14:25:50.648923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.648892 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:50.649100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.648956 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbt99\" (UniqueName: \"kubernetes.io/projected/dd844848-51b7-4c5c-9c4d-289622911ccc-kube-api-access-wbt99\") pod \"maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:50.649100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.649083 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:50.672192 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.672156 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:25:50.672192 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.672181 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:25:50.749846 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.749815 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:50.750046 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.749909 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:50.750046 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.749963 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbt99\" (UniqueName: \"kubernetes.io/projected/dd844848-51b7-4c5c-9c4d-289622911ccc-kube-api-access-wbt99\") pod \"maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:50.750798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.750769 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:50.752529 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.752505 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:50.772030 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.772004 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbt99\" (UniqueName: \"kubernetes.io/projected/dd844848-51b7-4c5c-9c4d-289622911ccc-kube-api-access-wbt99\") pod \"maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:50.841854 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.841818 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" podUID="13dedc53-2e29-4eae-9ba3-267ae50634b4" containerName="payload-processing" containerID="cri-o://e83d84ec2ca8891666231b06789e9a3a7599ae9e95bd2ac0163c076201d6f979" gracePeriod=25 Oct 30 14:25:50.911282 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.911252 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:50.950101 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.950037 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:25:50.952209 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.952187 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-kube-api-access-m9t67\" (UniqueName: \"kubernetes.io/projected/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-kube-api-access-m9t67\") pod \"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2\" (UID: \"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2\") " Oct 30 14:25:50.952300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.952271 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-plugins-config-volume\") pod \"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2\" (UID: \"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2\") " Oct 30 14:25:50.952630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.952607 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-plugins-config-volume" pod "3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2" (UID: "3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:50.953005 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.952983 4968 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:50.954400 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:50.954380 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-kube-api-access-m9t67" pod "3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2" (UID: "3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2"). InnerVolumeSpecName "kube-api-access-m9t67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:51.055024 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.054986 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9t67\" (UniqueName: \"kubernetes.io/projected/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2-kube-api-access-m9t67\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:51.229622 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.229589 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv"] Oct 30 14:25:51.231197 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:51.231173 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd844848_51b7_4c5c_9c4d_289622911ccc.slice/crio-b8a8724211fb5e2f7980bd03e62db6f3fc51817ea3598714cbbb30ba841d76fc WatchSource:0}: Error finding container b8a8724211fb5e2f7980bd03e62db6f3fc51817ea3598714cbbb30ba841d76fc: Status 404 returned error can't find the container with id b8a8724211fb5e2f7980bd03e62db6f3fc51817ea3598714cbbb30ba841d76fc Oct 30 14:25:51.551526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.551493 4968 generic.go:356] "Generic (PLEG): container finished" podID="3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2" containerID="287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a" exitCode=0 Oct 30 14:25:51.551695 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.551543 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" Oct 30 14:25:51.551695 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.551584 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" event={"ID":"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2","Type":"ContainerDied","Data":"287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a"} Oct 30 14:25:51.551695 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.551645 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb" event={"ID":"3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2","Type":"ContainerDied","Data":"00b75d49544803b2ae49cbcfbc523a86c986fd8d8b98a632f0413e97dc39d5a3"} Oct 30 14:25:51.551695 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.551677 4968 scope.go:122] "RemoveContainer" containerID="287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a" Oct 30 14:25:51.553479 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.553451 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" event={"ID":"dd844848-51b7-4c5c-9c4d-289622911ccc","Type":"ContainerStarted","Data":"e630481f115e90f816dce401ac5105217adc9c82ddb6215d5f1738a7cac4e4b2"} Oct 30 14:25:51.553583 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.553506 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" event={"ID":"dd844848-51b7-4c5c-9c4d-289622911ccc","Type":"ContainerStarted","Data":"b8a8724211fb5e2f7980bd03e62db6f3fc51817ea3598714cbbb30ba841d76fc"} Oct 30 14:25:51.563517 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.563502 4968 scope.go:122] "RemoveContainer" containerID="287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a" Oct 30 14:25:51.563880 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:25:51.563861 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a\": container with ID starting with 287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a not found: ID does not exist" containerID="287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a" Oct 30 14:25:51.563942 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.563887 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a"} err="failed to get container status \"287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a\": rpc error: code = NotFound desc = could not find container \"287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a\": container with ID starting with 287a76768aaf3125090395f1342db804532f0fa9f804588e0b6806d187346d0a not found: ID does not exist" Oct 30 14:25:51.612160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.612009 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" podStartSLOduration=1.611992139 podStartE2EDuration="1.611992139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:51.610116881 +0000 UTC m=+1147.235560251" watchObservedRunningTime="2026-10-30 14:25:51.611992139 +0000 UTC m=+1147.237435513" Oct 30 14:25:51.728180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.728141 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb"] Oct 30 14:25:51.755987 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:51.755821 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a5ddf3-79579dbb99-ckjcb"] Oct 30 14:25:52.062415 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.062338 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28"] Oct 30 14:25:52.099854 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.099824 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb"] Oct 30 14:25:52.406117 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.405746 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk"] Oct 30 14:25:52.407055 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.406786 4968 cpu_manager.go:414] "containerMap: removing container" podUID="3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2" containerName="payload-processing" Oct 30 14:25:52.407055 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.406811 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2" containerName="payload-processing" Oct 30 14:25:52.407055 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.407026 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2" containerName="payload-processing" Oct 30 14:25:52.428320 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.428281 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk"] Oct 30 14:25:52.428498 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.428415 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:52.498797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.498749 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb063a2-ntpjk\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:52.498989 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.498966 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2fz\" (UniqueName: \"kubernetes.io/projected/efb89876-ad55-431b-b338-e8b22313c4ab-kube-api-access-fw2fz\") pod \"maas-api-revoke-sub-cdb063a2-ntpjk\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:52.499046 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.499016 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb063a2-ntpjk\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:52.519669 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.519636 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2" path="/var/lib/kubelet/pods/3c2a455d-6c77-4a8f-a34f-5ce1de4c39e2/volumes" Oct 30 14:25:52.564123 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.564091 4968 generic.go:356] "Generic (PLEG): container finished" podID="dd844848-51b7-4c5c-9c4d-289622911ccc" containerID="e630481f115e90f816dce401ac5105217adc9c82ddb6215d5f1738a7cac4e4b2" exitCode=0 Oct 30 14:25:52.564267 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.564175 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" event={"ID":"dd844848-51b7-4c5c-9c4d-289622911ccc","Type":"ContainerDied","Data":"e630481f115e90f816dce401ac5105217adc9c82ddb6215d5f1738a7cac4e4b2"} Oct 30 14:25:52.600164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.600127 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2fz\" (UniqueName: \"kubernetes.io/projected/efb89876-ad55-431b-b338-e8b22313c4ab-kube-api-access-fw2fz\") pod \"maas-api-revoke-sub-cdb063a2-ntpjk\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:52.600325 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.600183 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb063a2-ntpjk\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:52.600325 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.600301 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb063a2-ntpjk\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:52.600987 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.600965 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb063a2-ntpjk\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:52.602896 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.602877 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb063a2-ntpjk\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:52.631940 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.631888 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2fz\" (UniqueName: \"kubernetes.io/projected/efb89876-ad55-431b-b338-e8b22313c4ab-kube-api-access-fw2fz\") pod \"maas-api-revoke-sub-cdb063a2-ntpjk\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:52.750039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:52.749963 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:53.282129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.282099 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk"] Oct 30 14:25:53.282476 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:53.282454 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb89876_ad55_431b_b338_e8b22313c4ab.slice/crio-21bdf5302a8d36cbbafb83482e59ff00736e13c4d4633149544f25263c4a031a WatchSource:0}: Error finding container 21bdf5302a8d36cbbafb83482e59ff00736e13c4d4633149544f25263c4a031a: Status 404 returned error can't find the container with id 21bdf5302a8d36cbbafb83482e59ff00736e13c4d4633149544f25263c4a031a Oct 30 14:25:53.573740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.573634 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" event={"ID":"efb89876-ad55-431b-b338-e8b22313c4ab","Type":"ContainerStarted","Data":"21bdf5302a8d36cbbafb83482e59ff00736e13c4d4633149544f25263c4a031a"} Oct 30 14:25:53.791937 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.791903 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:53.873982 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.873955 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-openshift-ca\") pod \"dd844848-51b7-4c5c-9c4d-289622911ccc\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " Oct 30 14:25:53.874142 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.874010 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd844848-51b7-4c5c-9c4d-289622911ccc-kube-api-access-wbt99\" (UniqueName: \"kubernetes.io/projected/dd844848-51b7-4c5c-9c4d-289622911ccc-kube-api-access-wbt99\") pod \"dd844848-51b7-4c5c-9c4d-289622911ccc\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " Oct 30 14:25:53.874142 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.874048 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-xks-ca\") pod \"dd844848-51b7-4c5c-9c4d-289622911ccc\" (UID: \"dd844848-51b7-4c5c-9c4d-289622911ccc\") " Oct 30 14:25:53.874307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.874277 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-openshift-ca" pod "dd844848-51b7-4c5c-9c4d-289622911ccc" (UID: "dd844848-51b7-4c5c-9c4d-289622911ccc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:53.874461 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.874444 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:53.876070 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.876048 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd844848-51b7-4c5c-9c4d-289622911ccc-kube-api-access-wbt99" pod "dd844848-51b7-4c5c-9c4d-289622911ccc" (UID: "dd844848-51b7-4c5c-9c4d-289622911ccc"). InnerVolumeSpecName "kube-api-access-wbt99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:53.876070 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.876062 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-xks-ca" pod "dd844848-51b7-4c5c-9c4d-289622911ccc" (UID: "dd844848-51b7-4c5c-9c4d-289622911ccc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:53.975693 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.975666 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbt99\" (UniqueName: \"kubernetes.io/projected/dd844848-51b7-4c5c-9c4d-289622911ccc-kube-api-access-wbt99\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:53.975693 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:53.975692 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd844848-51b7-4c5c-9c4d-289622911ccc-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:54.582537 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:54.582505 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" Oct 30 14:25:54.583017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:54.582531 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv" event={"ID":"dd844848-51b7-4c5c-9c4d-289622911ccc","Type":"ContainerDied","Data":"b8a8724211fb5e2f7980bd03e62db6f3fc51817ea3598714cbbb30ba841d76fc"} Oct 30 14:25:54.583017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:54.582586 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8a8724211fb5e2f7980bd03e62db6f3fc51817ea3598714cbbb30ba841d76fc" Oct 30 14:25:54.584631 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:54.584578 4968 generic.go:356] "Generic (PLEG): container finished" podID="efb89876-ad55-431b-b338-e8b22313c4ab" containerID="2846bac9b5b79b457c8901eac81f3741c455462246c6f5668f317259f9cd9793" exitCode=0 Oct 30 14:25:54.584787 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:54.584685 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" event={"ID":"efb89876-ad55-431b-b338-e8b22313c4ab","Type":"ContainerDied","Data":"2846bac9b5b79b457c8901eac81f3741c455462246c6f5668f317259f9cd9793"} Oct 30 14:25:55.236802 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.236763 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52"] Oct 30 14:25:55.237456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.237435 4968 cpu_manager.go:414] "containerMap: removing container" podUID="dd844848-51b7-4c5c-9c4d-289622911ccc" containerName="revoke-keys" Oct 30 14:25:55.237573 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.237457 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd844848-51b7-4c5c-9c4d-289622911ccc" containerName="revoke-keys" Oct 30 14:25:55.237638 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.237606 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd844848-51b7-4c5c-9c4d-289622911ccc" containerName="revoke-keys" Oct 30 14:25:55.241351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.241329 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:55.262901 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.261706 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52"] Oct 30 14:25:55.371520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.369210 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47849a1e-m6g52\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:55.371520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.369308 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9zq\" (UniqueName: \"kubernetes.io/projected/e41e256b-af47-49bb-a8a0-a9374935fab8-kube-api-access-qh9zq\") pod \"maas-api-revoke-sub-47849a1e-m6g52\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:55.371520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.369382 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47849a1e-m6g52\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:55.376086 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.376042 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" podUID="48066b06-babd-49b6-961a-1bf34b8c7973" containerName="main" containerID="cri-o://1228f776cc0f6d045f4d62a9ae07c784e791a175f445b1398b3955c549f0c6aa" gracePeriod=45 Oct 30 14:25:55.390619 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.390520 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" podUID="554e6fd9-60f6-49bb-a2a4-ef944cc98136" containerName="main" containerID="cri-o://e2cbd1e7c6870e96da50293d85f4dd63e3e76a13e8f7d65f138d87a9a8bdc427" gracePeriod=45 Oct 30 14:25:55.410313 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.409108 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8"] Oct 30 14:25:55.420777 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.418472 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd"] Oct 30 14:25:55.430315 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.428032 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" podUID="182fc12f-335a-4b5c-b26c-6ae65caa58cb" containerName="main" containerID="cri-o://f7535bf5b4b32715b40a1cab9587e8c5a14b23569a05daa901ddcaf8ce9b03eb" gracePeriod=45 Oct 30 14:25:55.471658 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.471629 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47849a1e-m6g52\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:55.471803 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.471687 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9zq\" (UniqueName: \"kubernetes.io/projected/e41e256b-af47-49bb-a8a0-a9374935fab8-kube-api-access-qh9zq\") pod \"maas-api-revoke-sub-47849a1e-m6g52\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:55.471803 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.471781 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47849a1e-m6g52\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:55.472437 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.472416 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47849a1e-m6g52\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:55.473838 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.473821 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47849a1e-m6g52\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:55.497773 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.497742 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9zq\" (UniqueName: \"kubernetes.io/projected/e41e256b-af47-49bb-a8a0-a9374935fab8-kube-api-access-qh9zq\") pod \"maas-api-revoke-sub-47849a1e-m6g52\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:55.557763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.555868 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:55.616867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.616815 4968 generic.go:356] "Generic (PLEG): container finished" podID="182fc12f-335a-4b5c-b26c-6ae65caa58cb" containerID="f7535bf5b4b32715b40a1cab9587e8c5a14b23569a05daa901ddcaf8ce9b03eb" exitCode=0 Oct 30 14:25:55.617349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.617013 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" event={"ID":"182fc12f-335a-4b5c-b26c-6ae65caa58cb","Type":"ContainerDied","Data":"f7535bf5b4b32715b40a1cab9587e8c5a14b23569a05daa901ddcaf8ce9b03eb"} Oct 30 14:25:55.619049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.618985 4968 generic.go:356] "Generic (PLEG): container finished" podID="48066b06-babd-49b6-961a-1bf34b8c7973" containerID="1228f776cc0f6d045f4d62a9ae07c784e791a175f445b1398b3955c549f0c6aa" exitCode=0 Oct 30 14:25:55.619181 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.619091 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" event={"ID":"48066b06-babd-49b6-961a-1bf34b8c7973","Type":"ContainerDied","Data":"1228f776cc0f6d045f4d62a9ae07c784e791a175f445b1398b3955c549f0c6aa"} Oct 30 14:25:55.621313 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.621287 4968 generic.go:356] "Generic (PLEG): container finished" podID="554e6fd9-60f6-49bb-a2a4-ef944cc98136" containerID="e2cbd1e7c6870e96da50293d85f4dd63e3e76a13e8f7d65f138d87a9a8bdc427" exitCode=0 Oct 30 14:25:55.621639 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.621607 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" event={"ID":"554e6fd9-60f6-49bb-a2a4-ef944cc98136","Type":"ContainerDied","Data":"e2cbd1e7c6870e96da50293d85f4dd63e3e76a13e8f7d65f138d87a9a8bdc427"} Oct 30 14:25:55.623189 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.623163 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" podUID="48066b06-babd-49b6-961a-1bf34b8c7973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Oct 30 14:25:55.822632 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.822610 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:25:55.826331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.826310 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:25:55.832468 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.832450 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.879923 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/554e6fd9-60f6-49bb-a2a4-ef944cc98136-kube-api-access-8867z\" (UniqueName: \"kubernetes.io/projected/554e6fd9-60f6-49bb-a2a4-ef944cc98136-kube-api-access-8867z\") pod \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880090 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-home\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-home\") pod \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880123 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-home\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-home\") pod \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880164 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tmp-dir\") pod \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880198 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-model-cache\") pod \"48066b06-babd-49b6-961a-1bf34b8c7973\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880227 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48066b06-babd-49b6-961a-1bf34b8c7973-kube-api-access-47nfv\" (UniqueName: \"kubernetes.io/projected/48066b06-babd-49b6-961a-1bf34b8c7973-kube-api-access-47nfv\") pod \"48066b06-babd-49b6-961a-1bf34b8c7973\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880256 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-tmp-dir\") pod \"48066b06-babd-49b6-961a-1bf34b8c7973\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880296 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182fc12f-335a-4b5c-b26c-6ae65caa58cb-kube-api-access-lpchv\" (UniqueName: \"kubernetes.io/projected/182fc12f-335a-4b5c-b26c-6ae65caa58cb-kube-api-access-lpchv\") pod \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880371 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tls-certs\" (UniqueName: \"kubernetes.io/secret/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tls-certs\") pod \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880441 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-dshm\") pod \"48066b06-babd-49b6-961a-1bf34b8c7973\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880504 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tmp-dir\") pod \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880567 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-dshm\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-dshm\") pod \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880614 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-model-cache\") pod \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880672 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tls-certs\") pod \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880699 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-dshm\") pod \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\" (UID: \"182fc12f-335a-4b5c-b26c-6ae65caa58cb\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880762 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-model-cache\") pod \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\" (UID: \"554e6fd9-60f6-49bb-a2a4-ef944cc98136\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880806 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48066b06-babd-49b6-961a-1bf34b8c7973-tls-certs\" (UniqueName: \"kubernetes.io/secret/48066b06-babd-49b6-961a-1bf34b8c7973-tls-certs\") pod \"48066b06-babd-49b6-961a-1bf34b8c7973\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " Oct 30 14:25:55.881588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.880842 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-home\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-home\") pod \"48066b06-babd-49b6-961a-1bf34b8c7973\" (UID: \"48066b06-babd-49b6-961a-1bf34b8c7973\") " Oct 30 14:25:55.882947 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.882654 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-home" pod "48066b06-babd-49b6-961a-1bf34b8c7973" (UID: "48066b06-babd-49b6-961a-1bf34b8c7973"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.884177 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.883865 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tmp-dir" pod "182fc12f-335a-4b5c-b26c-6ae65caa58cb" (UID: "182fc12f-335a-4b5c-b26c-6ae65caa58cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.884898 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.884643 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-model-cache" pod "182fc12f-335a-4b5c-b26c-6ae65caa58cb" (UID: "182fc12f-335a-4b5c-b26c-6ae65caa58cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.885175 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.885142 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-model-cache" pod "48066b06-babd-49b6-961a-1bf34b8c7973" (UID: "48066b06-babd-49b6-961a-1bf34b8c7973"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.885304 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.885288 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-tmp-dir" pod "48066b06-babd-49b6-961a-1bf34b8c7973" (UID: "48066b06-babd-49b6-961a-1bf34b8c7973"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.885405 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.885392 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-home" pod "182fc12f-335a-4b5c-b26c-6ae65caa58cb" (UID: "182fc12f-335a-4b5c-b26c-6ae65caa58cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.885517 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.885504 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-home" pod "554e6fd9-60f6-49bb-a2a4-ef944cc98136" (UID: "554e6fd9-60f6-49bb-a2a4-ef944cc98136"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.885647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.885634 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tmp-dir" pod "554e6fd9-60f6-49bb-a2a4-ef944cc98136" (UID: "554e6fd9-60f6-49bb-a2a4-ef944cc98136"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.887421 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.887400 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-model-cache" pod "554e6fd9-60f6-49bb-a2a4-ef944cc98136" (UID: "554e6fd9-60f6-49bb-a2a4-ef944cc98136"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.890258 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.890142 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:55.890993 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.890925 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-dshm" pod "48066b06-babd-49b6-961a-1bf34b8c7973" (UID: "48066b06-babd-49b6-961a-1bf34b8c7973"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.891184 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.891156 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554e6fd9-60f6-49bb-a2a4-ef944cc98136-kube-api-access-8867z" pod "554e6fd9-60f6-49bb-a2a4-ef944cc98136" (UID: "554e6fd9-60f6-49bb-a2a4-ef944cc98136"). InnerVolumeSpecName "kube-api-access-8867z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:55.893094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.893061 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-dshm" pod "182fc12f-335a-4b5c-b26c-6ae65caa58cb" (UID: "182fc12f-335a-4b5c-b26c-6ae65caa58cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.893453 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.893432 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-dshm" pod "554e6fd9-60f6-49bb-a2a4-ef944cc98136" (UID: "554e6fd9-60f6-49bb-a2a4-ef944cc98136"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:55.893650 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.893619 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48066b06-babd-49b6-961a-1bf34b8c7973-kube-api-access-47nfv" pod "48066b06-babd-49b6-961a-1bf34b8c7973" (UID: "48066b06-babd-49b6-961a-1bf34b8c7973"). InnerVolumeSpecName "kube-api-access-47nfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:55.893798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.893782 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48066b06-babd-49b6-961a-1bf34b8c7973-tls-certs" pod "48066b06-babd-49b6-961a-1bf34b8c7973" (UID: "48066b06-babd-49b6-961a-1bf34b8c7973"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:55.894021 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.893996 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tls-certs" pod "554e6fd9-60f6-49bb-a2a4-ef944cc98136" (UID: "554e6fd9-60f6-49bb-a2a4-ef944cc98136"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:55.895610 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.895578 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tls-certs" pod "182fc12f-335a-4b5c-b26c-6ae65caa58cb" (UID: "182fc12f-335a-4b5c-b26c-6ae65caa58cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:55.896533 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.896497 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182fc12f-335a-4b5c-b26c-6ae65caa58cb-kube-api-access-lpchv" pod "182fc12f-335a-4b5c-b26c-6ae65caa58cb" (UID: "182fc12f-335a-4b5c-b26c-6ae65caa58cb"). InnerVolumeSpecName "kube-api-access-lpchv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:55.980563 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:55.980531 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41e256b_af47_49bb_a8a0_a9374935fab8.slice/crio-ab8194ed782f788e5f35424167b00da1467504ba9a57f6f29bbd6e51eaa8742b WatchSource:0}: Error finding container ab8194ed782f788e5f35424167b00da1467504ba9a57f6f29bbd6e51eaa8742b: Status 404 returned error can't find the container with id ab8194ed782f788e5f35424167b00da1467504ba9a57f6f29bbd6e51eaa8742b Oct 30 14:25:55.983467 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.982201 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-xks-ca\") pod \"efb89876-ad55-431b-b338-e8b22313c4ab\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " Oct 30 14:25:55.983822 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.983782 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb89876-ad55-431b-b338-e8b22313c4ab-kube-api-access-fw2fz\" (UniqueName: \"kubernetes.io/projected/efb89876-ad55-431b-b338-e8b22313c4ab-kube-api-access-fw2fz\") pod \"efb89876-ad55-431b-b338-e8b22313c4ab\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " Oct 30 14:25:55.983931 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.983915 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-openshift-ca\") pod \"efb89876-ad55-431b-b338-e8b22313c4ab\" (UID: \"efb89876-ad55-431b-b338-e8b22313c4ab\") " Oct 30 14:25:55.984400 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984375 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984402 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984416 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984429 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984442 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47nfv\" (UniqueName: \"kubernetes.io/projected/48066b06-babd-49b6-961a-1bf34b8c7973-kube-api-access-47nfv\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984454 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984468 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpchv\" (UniqueName: \"kubernetes.io/projected/182fc12f-335a-4b5c-b26c-6ae65caa58cb-kube-api-access-lpchv\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984481 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984493 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984506 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984519 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984531 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984544 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182fc12f-335a-4b5c-b26c-6ae65caa58cb-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984555 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182fc12f-335a-4b5c-b26c-6ae65caa58cb-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984567 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554e6fd9-60f6-49bb-a2a4-ef944cc98136-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984581 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48066b06-babd-49b6-961a-1bf34b8c7973-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984594 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48066b06-babd-49b6-961a-1bf34b8c7973-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984607 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8867z\" (UniqueName: \"kubernetes.io/projected/554e6fd9-60f6-49bb-a2a4-ef944cc98136-kube-api-access-8867z\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:55.984980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.984956 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-openshift-ca" pod "efb89876-ad55-431b-b338-e8b22313c4ab" (UID: "efb89876-ad55-431b-b338-e8b22313c4ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:55.986773 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.986742 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52"] Oct 30 14:25:55.998662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.998636 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-xks-ca" pod "efb89876-ad55-431b-b338-e8b22313c4ab" (UID: "efb89876-ad55-431b-b338-e8b22313c4ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:55.998834 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:55.998668 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb89876-ad55-431b-b338-e8b22313c4ab-kube-api-access-fw2fz" pod "efb89876-ad55-431b-b338-e8b22313c4ab" (UID: "efb89876-ad55-431b-b338-e8b22313c4ab"). InnerVolumeSpecName "kube-api-access-fw2fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:56.090038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.085838 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:56.090038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.085877 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb89876-ad55-431b-b338-e8b22313c4ab-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:56.090038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.085894 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2fz\" (UniqueName: \"kubernetes.io/projected/efb89876-ad55-431b-b338-e8b22313c4ab-kube-api-access-fw2fz\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:56.631960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.631932 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" Oct 30 14:25:56.632382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.631955 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl" event={"ID":"48066b06-babd-49b6-961a-1bf34b8c7973","Type":"ContainerDied","Data":"10729ce8853ecd4c4da9a3f943a3a5fdfb70a9f148123e0a1ce6fa55b376b42f"} Oct 30 14:25:56.632382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.632008 4968 scope.go:122] "RemoveContainer" containerID="1228f776cc0f6d045f4d62a9ae07c784e791a175f445b1398b3955c549f0c6aa" Oct 30 14:25:56.633686 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.633671 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" Oct 30 14:25:56.633825 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.633739 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6" event={"ID":"554e6fd9-60f6-49bb-a2a4-ef944cc98136","Type":"ContainerDied","Data":"7613dcdbfb397ad35b53736ab2d2a5e512484ec08c02c493bce9c365da489688"} Oct 30 14:25:56.635779 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.635759 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" event={"ID":"efb89876-ad55-431b-b338-e8b22313c4ab","Type":"ContainerDied","Data":"21bdf5302a8d36cbbafb83482e59ff00736e13c4d4633149544f25263c4a031a"} Oct 30 14:25:56.635917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.635783 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk" Oct 30 14:25:56.635917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.635802 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21bdf5302a8d36cbbafb83482e59ff00736e13c4d4633149544f25263c4a031a" Oct 30 14:25:56.640640 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.640615 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" event={"ID":"182fc12f-335a-4b5c-b26c-6ae65caa58cb","Type":"ContainerDied","Data":"bdad96c5e24cb6d2cf2910bab68e3eb957a7c3245bfe637ecf991aa455f45aa7"} Oct 30 14:25:56.641561 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.641542 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" Oct 30 14:25:56.644403 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.644378 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" event={"ID":"e41e256b-af47-49bb-a8a0-a9374935fab8","Type":"ContainerStarted","Data":"a40b2b020512e57f6ebe4cd0de71c7edc2195e7e25690fb59bb963807416d0cd"} Oct 30 14:25:56.644502 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.644423 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" event={"ID":"e41e256b-af47-49bb-a8a0-a9374935fab8","Type":"ContainerStarted","Data":"ab8194ed782f788e5f35424167b00da1467504ba9a57f6f29bbd6e51eaa8742b"} Oct 30 14:25:56.646865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.646850 4968 scope.go:122] "RemoveContainer" containerID="e2cbd1e7c6870e96da50293d85f4dd63e3e76a13e8f7d65f138d87a9a8bdc427" Oct 30 14:25:56.655756 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.655735 4968 scope.go:122] "RemoveContainer" containerID="f7535bf5b4b32715b40a1cab9587e8c5a14b23569a05daa901ddcaf8ce9b03eb" Oct 30 14:25:56.697764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.697667 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" podStartSLOduration=1.6976526 podStartE2EDuration="1.6976526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:56.696868196 +0000 UTC m=+1152.322311572" watchObservedRunningTime="2026-10-30 14:25:56.6976526 +0000 UTC m=+1152.323095974" Oct 30 14:25:56.731941 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.731911 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6"] Oct 30 14:25:56.738977 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.738951 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6"] Oct 30 14:25:56.761941 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.761900 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-24dfd7/simulator-w3-24dfd7-kserve-868784b9c7-m5bl6" podUID="182fc12f-335a-4b5c-b26c-6ae65caa58cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 14:25:56.770055 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.770033 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl"] Oct 30 14:25:56.774872 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.774850 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-w3-24dfd7-kserve-784f985c5-c9bkl"] Oct 30 14:25:56.805626 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.805602 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6"] Oct 30 14:25:56.812008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:56.811982 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-24dfd7/distinct-2-w3-24dfd7-kserve-75bb47fb6d-xhtg6"] Oct 30 14:25:57.063654 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:57.063547 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" podUID="0686340e-12ce-45b1-8135-2cf61f0a3f0b" containerName="payload-processing" containerID="cri-o://a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666" gracePeriod=25 Oct 30 14:25:57.131749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:57.128796 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" podUID="b3ea3629-deb5-427f-849b-2e40f1bbabc5" containerName="payload-pre-processing" containerID="cri-o://88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a" gracePeriod=25 Oct 30 14:25:57.243063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:57.243031 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" podUID="13dedc53-2e29-4eae-9ba3-267ae50634b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:25:57.655099 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:57.655062 4968 generic.go:356] "Generic (PLEG): container finished" podID="e41e256b-af47-49bb-a8a0-a9374935fab8" containerID="a40b2b020512e57f6ebe4cd0de71c7edc2195e7e25690fb59bb963807416d0cd" exitCode=0 Oct 30 14:25:57.655678 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:57.655103 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" event={"ID":"e41e256b-af47-49bb-a8a0-a9374935fab8","Type":"ContainerDied","Data":"a40b2b020512e57f6ebe4cd0de71c7edc2195e7e25690fb59bb963807416d0cd"} Oct 30 14:25:58.499709 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.499667 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r"] Oct 30 14:25:58.500313 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500295 4968 cpu_manager.go:414] "containerMap: removing container" podUID="554e6fd9-60f6-49bb-a2a4-ef944cc98136" containerName="main" Oct 30 14:25:58.500313 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500313 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554e6fd9-60f6-49bb-a2a4-ef944cc98136" containerName="main" Oct 30 14:25:58.500442 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500334 4968 cpu_manager.go:414] "containerMap: removing container" podUID="48066b06-babd-49b6-961a-1bf34b8c7973" containerName="main" Oct 30 14:25:58.500442 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500339 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48066b06-babd-49b6-961a-1bf34b8c7973" containerName="main" Oct 30 14:25:58.500442 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500364 4968 cpu_manager.go:414] "containerMap: removing container" podUID="182fc12f-335a-4b5c-b26c-6ae65caa58cb" containerName="main" Oct 30 14:25:58.500442 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500370 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182fc12f-335a-4b5c-b26c-6ae65caa58cb" containerName="main" Oct 30 14:25:58.500442 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500380 4968 cpu_manager.go:414] "containerMap: removing container" podUID="efb89876-ad55-431b-b338-e8b22313c4ab" containerName="revoke-keys" Oct 30 14:25:58.500442 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500384 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb89876-ad55-431b-b338-e8b22313c4ab" containerName="revoke-keys" Oct 30 14:25:58.500765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500477 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb89876-ad55-431b-b338-e8b22313c4ab" containerName="revoke-keys" Oct 30 14:25:58.500765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500488 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="554e6fd9-60f6-49bb-a2a4-ef944cc98136" containerName="main" Oct 30 14:25:58.500765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500496 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="48066b06-babd-49b6-961a-1bf34b8c7973" containerName="main" Oct 30 14:25:58.500765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.500506 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="182fc12f-335a-4b5c-b26c-6ae65caa58cb" containerName="main" Oct 30 14:25:58.504503 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.504481 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:25:58.522063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.522036 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182fc12f-335a-4b5c-b26c-6ae65caa58cb" path="/var/lib/kubelet/pods/182fc12f-335a-4b5c-b26c-6ae65caa58cb/volumes" Oct 30 14:25:58.522905 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.522877 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48066b06-babd-49b6-961a-1bf34b8c7973" path="/var/lib/kubelet/pods/48066b06-babd-49b6-961a-1bf34b8c7973/volumes" Oct 30 14:25:58.523488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.523465 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="554e6fd9-60f6-49bb-a2a4-ef944cc98136" path="/var/lib/kubelet/pods/554e6fd9-60f6-49bb-a2a4-ef944cc98136/volumes" Oct 30 14:25:58.528104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.528080 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r"] Oct 30 14:25:58.564130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.564098 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9dcb4e-qcc8r\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:25:58.564299 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.564250 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9dcb4e-qcc8r\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:25:58.564390 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.564365 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2k45\" (UniqueName: \"kubernetes.io/projected/e2b66c2f-7a81-4340-a5ed-c539605c2389-kube-api-access-v2k45\") pod \"maas-api-revoke-sub-8e9dcb4e-qcc8r\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:25:58.665763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.665699 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2k45\" (UniqueName: \"kubernetes.io/projected/e2b66c2f-7a81-4340-a5ed-c539605c2389-kube-api-access-v2k45\") pod \"maas-api-revoke-sub-8e9dcb4e-qcc8r\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:25:58.666246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.665844 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9dcb4e-qcc8r\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:25:58.666246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.665914 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9dcb4e-qcc8r\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:25:58.666662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.666538 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9dcb4e-qcc8r\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:25:58.668484 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.668459 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9dcb4e-qcc8r\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:25:58.698640 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.698604 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2k45\" (UniqueName: \"kubernetes.io/projected/e2b66c2f-7a81-4340-a5ed-c539605c2389-kube-api-access-v2k45\") pod \"maas-api-revoke-sub-8e9dcb4e-qcc8r\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:25:58.820087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.820057 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:25:58.857863 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.857839 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:58.976434 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.972171 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-openshift-ca\") pod \"e41e256b-af47-49bb-a8a0-a9374935fab8\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " Oct 30 14:25:58.976434 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.972284 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-xks-ca\") pod \"e41e256b-af47-49bb-a8a0-a9374935fab8\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " Oct 30 14:25:58.976434 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.972333 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41e256b-af47-49bb-a8a0-a9374935fab8-kube-api-access-qh9zq\" (UniqueName: \"kubernetes.io/projected/e41e256b-af47-49bb-a8a0-a9374935fab8-kube-api-access-qh9zq\") pod \"e41e256b-af47-49bb-a8a0-a9374935fab8\" (UID: \"e41e256b-af47-49bb-a8a0-a9374935fab8\") " Oct 30 14:25:58.976434 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.973653 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-openshift-ca" pod "e41e256b-af47-49bb-a8a0-a9374935fab8" (UID: "e41e256b-af47-49bb-a8a0-a9374935fab8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:58.977110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:58.976648 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-xks-ca" pod "e41e256b-af47-49bb-a8a0-a9374935fab8" (UID: "e41e256b-af47-49bb-a8a0-a9374935fab8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:59.011577 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.011541 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41e256b-af47-49bb-a8a0-a9374935fab8-kube-api-access-qh9zq" pod "e41e256b-af47-49bb-a8a0-a9374935fab8" (UID: "e41e256b-af47-49bb-a8a0-a9374935fab8"). InnerVolumeSpecName "kube-api-access-qh9zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:59.074441 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.074365 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:59.074441 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.074394 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41e256b-af47-49bb-a8a0-a9374935fab8-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:59.074441 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.074403 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh9zq\" (UniqueName: \"kubernetes.io/projected/e41e256b-af47-49bb-a8a0-a9374935fab8-kube-api-access-qh9zq\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:25:59.138154 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:25:59.138120 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b66c2f_7a81_4340_a5ed_c539605c2389.slice/crio-1c01b9e6cfbf55f9ad61a85a55c8e08709d016502a26c974c4dc37c63408211c WatchSource:0}: Error finding container 1c01b9e6cfbf55f9ad61a85a55c8e08709d016502a26c974c4dc37c63408211c: Status 404 returned error can't find the container with id 1c01b9e6cfbf55f9ad61a85a55c8e08709d016502a26c974c4dc37c63408211c Oct 30 14:25:59.138154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.138138 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r"] Oct 30 14:25:59.677482 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.677399 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" event={"ID":"e2b66c2f-7a81-4340-a5ed-c539605c2389","Type":"ContainerStarted","Data":"5988ce60f65a43ca53cfd7f5322d8fedbb96403a1cf056334e92cce120b3c1de"} Oct 30 14:25:59.677482 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.677463 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" event={"ID":"e2b66c2f-7a81-4340-a5ed-c539605c2389","Type":"ContainerStarted","Data":"1c01b9e6cfbf55f9ad61a85a55c8e08709d016502a26c974c4dc37c63408211c"} Oct 30 14:25:59.679369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.679338 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" event={"ID":"e41e256b-af47-49bb-a8a0-a9374935fab8","Type":"ContainerDied","Data":"ab8194ed782f788e5f35424167b00da1467504ba9a57f6f29bbd6e51eaa8742b"} Oct 30 14:25:59.679510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.679391 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab8194ed782f788e5f35424167b00da1467504ba9a57f6f29bbd6e51eaa8742b" Oct 30 14:25:59.679510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.679454 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52" Oct 30 14:25:59.720686 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.720622 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" podStartSLOduration=1.720602585 podStartE2EDuration="1.720602585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:59.714655014 +0000 UTC m=+1155.340098385" watchObservedRunningTime="2026-10-30 14:25:59.720602585 +0000 UTC m=+1155.346045961" Oct 30 14:25:59.787961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.787870 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s"] Oct 30 14:25:59.788517 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.788226 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" containerID="cri-o://dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d" gracePeriod=30 Oct 30 14:25:59.801831 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.801566 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62"] Oct 30 14:25:59.802005 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:25:59.801904 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" podUID="39235b85-adb6-477d-b069-64e29e05fd17" containerName="istio-proxy" containerID="cri-o://2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c" gracePeriod=30 Oct 30 14:26:00.264225 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.264179 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" podUID="b3ea3629-deb5-427f-849b-2e40f1bbabc5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:26:00.309415 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.309385 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk"] Oct 30 14:26:00.310100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.310083 4968 cpu_manager.go:414] "containerMap: removing container" podUID="e41e256b-af47-49bb-a8a0-a9374935fab8" containerName="revoke-keys" Oct 30 14:26:00.310159 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.310102 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41e256b-af47-49bb-a8a0-a9374935fab8" containerName="revoke-keys" Oct 30 14:26:00.310246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.310236 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41e256b-af47-49bb-a8a0-a9374935fab8" containerName="revoke-keys" Oct 30 14:26:00.313904 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.313889 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:00.332114 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.332042 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj"] Oct 30 14:26:00.337438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.337417 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:00.337973 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.337953 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk"] Oct 30 14:26:00.359671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.359646 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj"] Oct 30 14:26:00.402224 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.402192 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e4c39a-brnvk\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:00.402366 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.402310 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e4c39a-brnvk\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:00.402467 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.402445 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5dj\" (UniqueName: \"kubernetes.io/projected/9d34ecd5-3570-487d-b86c-d315737475d5-kube-api-access-jp5dj\") pod \"maas-api-revoke-sub-fd21d625-b9kdj\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:00.402634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.402606 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6s7p\" (UniqueName: \"kubernetes.io/projected/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-kube-api-access-v6s7p\") pod \"maas-api-revoke-sub-e3e4c39a-brnvk\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:00.402704 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.402664 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd21d625-b9kdj\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:00.402819 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.402701 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd21d625-b9kdj\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:00.504010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.503973 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5dj\" (UniqueName: \"kubernetes.io/projected/9d34ecd5-3570-487d-b86c-d315737475d5-kube-api-access-jp5dj\") pod \"maas-api-revoke-sub-fd21d625-b9kdj\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:00.504183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.504045 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6s7p\" (UniqueName: \"kubernetes.io/projected/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-kube-api-access-v6s7p\") pod \"maas-api-revoke-sub-e3e4c39a-brnvk\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:00.504183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.504103 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd21d625-b9kdj\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:00.504183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.504136 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd21d625-b9kdj\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:00.504183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.504172 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e4c39a-brnvk\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:00.504428 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.504285 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e4c39a-brnvk\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:00.505025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.504993 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e4c39a-brnvk\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:00.505341 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.505315 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd21d625-b9kdj\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:00.507279 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.507257 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e4c39a-brnvk\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:00.507397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.507301 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd21d625-b9kdj\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:00.527818 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.527755 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5dj\" (UniqueName: \"kubernetes.io/projected/9d34ecd5-3570-487d-b86c-d315737475d5-kube-api-access-jp5dj\") pod \"maas-api-revoke-sub-fd21d625-b9kdj\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:00.527818 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.527755 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6s7p\" (UniqueName: \"kubernetes.io/projected/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-kube-api-access-v6s7p\") pod \"maas-api-revoke-sub-e3e4c39a-brnvk\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:00.678328 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.678292 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:00.687450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.687422 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:00.689066 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.689042 4968 generic.go:356] "Generic (PLEG): container finished" podID="e2b66c2f-7a81-4340-a5ed-c539605c2389" containerID="5988ce60f65a43ca53cfd7f5322d8fedbb96403a1cf056334e92cce120b3c1de" exitCode=0 Oct 30 14:26:00.689183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:00.689174 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" event={"ID":"e2b66c2f-7a81-4340-a5ed-c539605c2389","Type":"ContainerDied","Data":"5988ce60f65a43ca53cfd7f5322d8fedbb96403a1cf056334e92cce120b3c1de"} Oct 30 14:26:01.005315 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:01.005121 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk"] Oct 30 14:26:01.024842 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:01.024818 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj"] Oct 30 14:26:01.030854 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:26:01.030819 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d34ecd5_3570_487d_b86c_d315737475d5.slice/crio-15577eb3be13be042f58760f50d22a2a810b595efd4686e0d57b18b106cc1b42 WatchSource:0}: Error finding container 15577eb3be13be042f58760f50d22a2a810b595efd4686e0d57b18b106cc1b42: Status 404 returned error can't find the container with id 15577eb3be13be042f58760f50d22a2a810b595efd4686e0d57b18b106cc1b42 Oct 30 14:26:01.073158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:01.073131 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9"] Oct 30 14:26:01.184994 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:01.184919 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:26:01.715795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:01.715756 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" event={"ID":"2b9c05ff-d1c5-4e2a-80b7-143b4757e660","Type":"ContainerStarted","Data":"fb08b4ba2795730cbca0fb7dae668902d25e3c5da3de0075e690f753437d2d95"} Oct 30 14:26:01.718841 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:01.715820 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" event={"ID":"2b9c05ff-d1c5-4e2a-80b7-143b4757e660","Type":"ContainerStarted","Data":"c2c13a03ede6330913b69555f725b52040f0ea706b45059f4bb7e337418b8865"} Oct 30 14:26:01.719619 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:01.719587 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" event={"ID":"9d34ecd5-3570-487d-b86c-d315737475d5","Type":"ContainerStarted","Data":"62247eb77e8560d6b8f4c4f66c0959e10d8751839f20d53e959da52b75b3707f"} Oct 30 14:26:01.719768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:01.719638 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" event={"ID":"9d34ecd5-3570-487d-b86c-d315737475d5","Type":"ContainerStarted","Data":"15577eb3be13be042f58760f50d22a2a810b595efd4686e0d57b18b106cc1b42"} Oct 30 14:26:01.760097 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:01.760045 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" podStartSLOduration=1.760029473 podStartE2EDuration="1.760029473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:01.759441397 +0000 UTC m=+1157.384884772" watchObservedRunningTime="2026-10-30 14:26:01.760029473 +0000 UTC m=+1157.385472845" Oct 30 14:26:01.802743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:01.801067 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" podStartSLOduration=1.801052152 podStartE2EDuration="1.801052152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:01.79767836 +0000 UTC m=+1157.423121733" watchObservedRunningTime="2026-10-30 14:26:01.801052152 +0000 UTC m=+1157.426495525" Oct 30 14:26:02.314745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.305876 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" podUID="0686340e-12ce-45b1-8135-2cf61f0a3f0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:26:02.410425 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.409928 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:26:02.501767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.491333 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-openshift-ca\") pod \"e2b66c2f-7a81-4340-a5ed-c539605c2389\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " Oct 30 14:26:02.510004 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.509971 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-xks-ca\") pod \"e2b66c2f-7a81-4340-a5ed-c539605c2389\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " Oct 30 14:26:02.510344 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.510319 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b66c2f-7a81-4340-a5ed-c539605c2389-kube-api-access-v2k45\" (UniqueName: \"kubernetes.io/projected/e2b66c2f-7a81-4340-a5ed-c539605c2389-kube-api-access-v2k45\") pod \"e2b66c2f-7a81-4340-a5ed-c539605c2389\" (UID: \"e2b66c2f-7a81-4340-a5ed-c539605c2389\") " Oct 30 14:26:02.515041 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.501791 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-openshift-ca" pod "e2b66c2f-7a81-4340-a5ed-c539605c2389" (UID: "e2b66c2f-7a81-4340-a5ed-c539605c2389"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:02.516390 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.516367 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:02.517973 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.517930 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b66c2f-7a81-4340-a5ed-c539605c2389-kube-api-access-v2k45" pod "e2b66c2f-7a81-4340-a5ed-c539605c2389" (UID: "e2b66c2f-7a81-4340-a5ed-c539605c2389"). InnerVolumeSpecName "kube-api-access-v2k45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:02.528340 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.528312 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-xks-ca" pod "e2b66c2f-7a81-4340-a5ed-c539605c2389" (UID: "e2b66c2f-7a81-4340-a5ed-c539605c2389"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:26:02.618592 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.618561 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2k45\" (UniqueName: \"kubernetes.io/projected/e2b66c2f-7a81-4340-a5ed-c539605c2389-kube-api-access-v2k45\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:02.618864 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.618836 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b66c2f-7a81-4340-a5ed-c539605c2389-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:02.861741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.858394 4968 generic.go:356] "Generic (PLEG): container finished" podID="2b9c05ff-d1c5-4e2a-80b7-143b4757e660" containerID="fb08b4ba2795730cbca0fb7dae668902d25e3c5da3de0075e690f753437d2d95" exitCode=0 Oct 30 14:26:02.861741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.858619 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" event={"ID":"2b9c05ff-d1c5-4e2a-80b7-143b4757e660","Type":"ContainerDied","Data":"fb08b4ba2795730cbca0fb7dae668902d25e3c5da3de0075e690f753437d2d95"} Oct 30 14:26:02.861741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.861284 4968 generic.go:356] "Generic (PLEG): container finished" podID="9d34ecd5-3570-487d-b86c-d315737475d5" containerID="62247eb77e8560d6b8f4c4f66c0959e10d8751839f20d53e959da52b75b3707f" exitCode=0 Oct 30 14:26:02.861741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.861465 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" event={"ID":"9d34ecd5-3570-487d-b86c-d315737475d5","Type":"ContainerDied","Data":"62247eb77e8560d6b8f4c4f66c0959e10d8751839f20d53e959da52b75b3707f"} Oct 30 14:26:02.869742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.863515 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" event={"ID":"e2b66c2f-7a81-4340-a5ed-c539605c2389","Type":"ContainerDied","Data":"1c01b9e6cfbf55f9ad61a85a55c8e08709d016502a26c974c4dc37c63408211c"} Oct 30 14:26:02.869742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.863557 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c01b9e6cfbf55f9ad61a85a55c8e08709d016502a26c974c4dc37c63408211c" Oct 30 14:26:02.869742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:02.863609 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r" Oct 30 14:26:03.578738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:03.577775 4968 patch_prober.go:29] interesting pod/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:26:03.578738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:03.577840 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:26:04.174697 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.173082 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9"] Oct 30 14:26:04.174697 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.174068 4968 cpu_manager.go:414] "containerMap: removing container" podUID="e2b66c2f-7a81-4340-a5ed-c539605c2389" containerName="revoke-keys" Oct 30 14:26:04.174697 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.174089 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b66c2f-7a81-4340-a5ed-c539605c2389" containerName="revoke-keys" Oct 30 14:26:04.174697 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.174240 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b66c2f-7a81-4340-a5ed-c539605c2389" containerName="revoke-keys" Oct 30 14:26:04.209917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.181979 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:04.269610 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.269582 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9"] Oct 30 14:26:04.360432 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.360387 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qvv\" (UniqueName: \"kubernetes.io/projected/bcbbc581-0989-460d-b26d-2dc65b74974e-kube-api-access-79qvv\") pod \"maas-api-revoke-sub-0e945de7-q9sd9\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:04.360609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.360511 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e945de7-q9sd9\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:04.360678 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.360606 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e945de7-q9sd9\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:04.371263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.371235 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:04.462245 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.462183 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-xks-ca\") pod \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " Oct 30 14:26:04.462382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.462243 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-kube-api-access-v6s7p\" (UniqueName: \"kubernetes.io/projected/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-kube-api-access-v6s7p\") pod \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " Oct 30 14:26:04.462382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.462275 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-openshift-ca\") pod \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\" (UID: \"2b9c05ff-d1c5-4e2a-80b7-143b4757e660\") " Oct 30 14:26:04.462453 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.462419 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e945de7-q9sd9\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:04.462504 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.462491 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qvv\" (UniqueName: \"kubernetes.io/projected/bcbbc581-0989-460d-b26d-2dc65b74974e-kube-api-access-79qvv\") pod \"maas-api-revoke-sub-0e945de7-q9sd9\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:04.462588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.462577 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e945de7-q9sd9\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:04.463074 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.463051 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:04.463763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.463301 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-openshift-ca" pod "2b9c05ff-d1c5-4e2a-80b7-143b4757e660" (UID: "2b9c05ff-d1c5-4e2a-80b7-143b4757e660"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:04.464250 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.464230 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e945de7-q9sd9\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:04.465967 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.465938 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e945de7-q9sd9\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:04.466294 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.466273 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-xks-ca" pod "2b9c05ff-d1c5-4e2a-80b7-143b4757e660" (UID: "2b9c05ff-d1c5-4e2a-80b7-143b4757e660"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:26:04.466943 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.466919 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-kube-api-access-v6s7p" pod "2b9c05ff-d1c5-4e2a-80b7-143b4757e660" (UID: "2b9c05ff-d1c5-4e2a-80b7-143b4757e660"). InnerVolumeSpecName "kube-api-access-v6s7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:04.487346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.487311 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qvv\" (UniqueName: \"kubernetes.io/projected/bcbbc581-0989-460d-b26d-2dc65b74974e-kube-api-access-79qvv\") pod \"maas-api-revoke-sub-0e945de7-q9sd9\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:04.563063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.563028 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d34ecd5-3570-487d-b86c-d315737475d5-kube-api-access-jp5dj\" (UniqueName: \"kubernetes.io/projected/9d34ecd5-3570-487d-b86c-d315737475d5-kube-api-access-jp5dj\") pod \"9d34ecd5-3570-487d-b86c-d315737475d5\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " Oct 30 14:26:04.563251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.563156 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-openshift-ca\") pod \"9d34ecd5-3570-487d-b86c-d315737475d5\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " Oct 30 14:26:04.563251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.563221 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-xks-ca\") pod \"9d34ecd5-3570-487d-b86c-d315737475d5\" (UID: \"9d34ecd5-3570-487d-b86c-d315737475d5\") " Oct 30 14:26:04.563546 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.563517 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-openshift-ca" pod "9d34ecd5-3570-487d-b86c-d315737475d5" (UID: "9d34ecd5-3570-487d-b86c-d315737475d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:04.563769 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.563751 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:04.563855 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.563770 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:04.563855 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.563784 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6s7p\" (UniqueName: \"kubernetes.io/projected/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-kube-api-access-v6s7p\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:04.563855 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.563796 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c05ff-d1c5-4e2a-80b7-143b4757e660-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:04.565367 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.565340 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-xks-ca" pod "9d34ecd5-3570-487d-b86c-d315737475d5" (UID: "9d34ecd5-3570-487d-b86c-d315737475d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:26:04.565474 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.565452 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d34ecd5-3570-487d-b86c-d315737475d5-kube-api-access-jp5dj" pod "9d34ecd5-3570-487d-b86c-d315737475d5" (UID: "9d34ecd5-3570-487d-b86c-d315737475d5"). InnerVolumeSpecName "kube-api-access-jp5dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:04.640044 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.640012 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:04.665326 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.665291 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp5dj\" (UniqueName: \"kubernetes.io/projected/9d34ecd5-3570-487d-b86c-d315737475d5-kube-api-access-jp5dj\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:04.665326 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.665322 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34ecd5-3570-487d-b86c-d315737475d5-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:04.904236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.902354 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" event={"ID":"2b9c05ff-d1c5-4e2a-80b7-143b4757e660","Type":"ContainerDied","Data":"c2c13a03ede6330913b69555f725b52040f0ea706b45059f4bb7e337418b8865"} Oct 30 14:26:04.904236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.902445 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c13a03ede6330913b69555f725b52040f0ea706b45059f4bb7e337418b8865" Oct 30 14:26:04.904236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.902562 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk" Oct 30 14:26:04.910231 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.910205 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" Oct 30 14:26:04.913608 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.913574 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj" event={"ID":"9d34ecd5-3570-487d-b86c-d315737475d5","Type":"ContainerDied","Data":"15577eb3be13be042f58760f50d22a2a810b595efd4686e0d57b18b106cc1b42"} Oct 30 14:26:04.913762 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.913644 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15577eb3be13be042f58760f50d22a2a810b595efd4686e0d57b18b106cc1b42" Oct 30 14:26:04.969798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:04.969700 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9"] Oct 30 14:26:04.969927 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:26:04.969882 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbbc581_0989_460d_b26d_2dc65b74974e.slice/crio-3451bccefbd5ff0e42bfc667ed2f107c74932677d5f4069ef63ba49ef2c70873 WatchSource:0}: Error finding container 3451bccefbd5ff0e42bfc667ed2f107c74932677d5f4069ef63ba49ef2c70873: Status 404 returned error can't find the container with id 3451bccefbd5ff0e42bfc667ed2f107c74932677d5f4069ef63ba49ef2c70873 Oct 30 14:26:05.192089 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.192060 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:26:05.194995 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.194970 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:26:05.271598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.271513 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b202666a-2b81-4d03-b286-a05d7a87b373-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b202666a-2b81-4d03-b286-a05d7a87b373-istiod-ca-cert\") pod \"b202666a-2b81-4d03-b286-a05d7a87b373\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " Oct 30 14:26:05.271795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.271603 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b202666a-2b81-4d03-b286-a05d7a87b373-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b202666a-2b81-4d03-b286-a05d7a87b373-istio-podinfo\") pod \"b202666a-2b81-4d03-b286-a05d7a87b373\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " Oct 30 14:26:05.271795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.271676 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-envoy\") pod \"39235b85-adb6-477d-b069-64e29e05fd17\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " Oct 30 14:26:05.271795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.271706 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-credential-socket\") pod \"39235b85-adb6-477d-b069-64e29e05fd17\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " Oct 30 14:26:05.271795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.271768 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-data\") pod \"39235b85-adb6-477d-b069-64e29e05fd17\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " Oct 30 14:26:05.272061 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.271863 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-istio-token\" (UniqueName: \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-istio-token\") pod \"b202666a-2b81-4d03-b286-a05d7a87b373\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " Oct 30 14:26:05.272061 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.271919 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-data\") pod \"b202666a-2b81-4d03-b286-a05d7a87b373\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " Oct 30 14:26:05.272061 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.271951 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-istio-token\" (UniqueName: \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-istio-token\") pod \"39235b85-adb6-477d-b069-64e29e05fd17\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " Oct 30 14:26:05.272061 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.271999 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-kube-api-access-4jzvp\" (UniqueName: \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-kube-api-access-4jzvp\") pod \"b202666a-2b81-4d03-b286-a05d7a87b373\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " Oct 30 14:26:05.272061 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272014 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b202666a-2b81-4d03-b286-a05d7a87b373-istiod-ca-cert" pod "b202666a-2b81-4d03-b286-a05d7a87b373" (UID: "b202666a-2b81-4d03-b286-a05d7a87b373"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:05.272061 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272032 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39235b85-adb6-477d-b069-64e29e05fd17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39235b85-adb6-477d-b069-64e29e05fd17-istio-podinfo\") pod \"39235b85-adb6-477d-b069-64e29e05fd17\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " Oct 30 14:26:05.272376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272078 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-credential-socket\") pod \"b202666a-2b81-4d03-b286-a05d7a87b373\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " Oct 30 14:26:05.272376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272141 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-certs\") pod \"39235b85-adb6-477d-b069-64e29e05fd17\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " Oct 30 14:26:05.272376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272200 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-envoy\") pod \"b202666a-2b81-4d03-b286-a05d7a87b373\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " Oct 30 14:26:05.272376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272238 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-socket\") pod \"39235b85-adb6-477d-b069-64e29e05fd17\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " Oct 30 14:26:05.272376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272218 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-data" pod "b202666a-2b81-4d03-b286-a05d7a87b373" (UID: "b202666a-2b81-4d03-b286-a05d7a87b373"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:05.272376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272267 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39235b85-adb6-477d-b069-64e29e05fd17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39235b85-adb6-477d-b069-64e29e05fd17-istiod-ca-cert\") pod \"39235b85-adb6-477d-b069-64e29e05fd17\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " Oct 30 14:26:05.272376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272278 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-data" pod "39235b85-adb6-477d-b069-64e29e05fd17" (UID: "39235b85-adb6-477d-b069-64e29e05fd17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:05.272376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272302 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-socket\") pod \"b202666a-2b81-4d03-b286-a05d7a87b373\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " Oct 30 14:26:05.272376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272350 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-certs\") pod \"b202666a-2b81-4d03-b286-a05d7a87b373\" (UID: \"b202666a-2b81-4d03-b286-a05d7a87b373\") " Oct 30 14:26:05.272966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272413 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-kube-api-access-4z2nl\" (UniqueName: \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-kube-api-access-4z2nl\") pod \"39235b85-adb6-477d-b069-64e29e05fd17\" (UID: \"39235b85-adb6-477d-b069-64e29e05fd17\") " Oct 30 14:26:05.272966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272704 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39235b85-adb6-477d-b069-64e29e05fd17-istiod-ca-cert" pod "39235b85-adb6-477d-b069-64e29e05fd17" (UID: "39235b85-adb6-477d-b069-64e29e05fd17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:05.272966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.272946 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-socket" pod "b202666a-2b81-4d03-b286-a05d7a87b373" (UID: "b202666a-2b81-4d03-b286-a05d7a87b373"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:05.273204 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.273180 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39235b85-adb6-477d-b069-64e29e05fd17-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.273311 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.273208 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.273311 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.273225 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b202666a-2b81-4d03-b286-a05d7a87b373-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.273311 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.273239 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.273311 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.273252 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.273311 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.273295 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-credential-socket" pod "39235b85-adb6-477d-b069-64e29e05fd17" (UID: "39235b85-adb6-477d-b069-64e29e05fd17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:05.273620 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.273599 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-credential-socket" pod "b202666a-2b81-4d03-b286-a05d7a87b373" (UID: "b202666a-2b81-4d03-b286-a05d7a87b373"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:05.273827 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.273805 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-socket" pod "39235b85-adb6-477d-b069-64e29e05fd17" (UID: "39235b85-adb6-477d-b069-64e29e05fd17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:05.273998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.273977 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-certs" pod "b202666a-2b81-4d03-b286-a05d7a87b373" (UID: "b202666a-2b81-4d03-b286-a05d7a87b373"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:05.274369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.274310 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-certs" pod "39235b85-adb6-477d-b069-64e29e05fd17" (UID: "39235b85-adb6-477d-b069-64e29e05fd17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:05.274483 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.274412 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b202666a-2b81-4d03-b286-a05d7a87b373-istio-podinfo" pod "b202666a-2b81-4d03-b286-a05d7a87b373" (UID: "b202666a-2b81-4d03-b286-a05d7a87b373"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:05.275223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.275188 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-envoy" pod "b202666a-2b81-4d03-b286-a05d7a87b373" (UID: "b202666a-2b81-4d03-b286-a05d7a87b373"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:05.275414 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.275393 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-envoy" pod "39235b85-adb6-477d-b069-64e29e05fd17" (UID: "39235b85-adb6-477d-b069-64e29e05fd17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:05.275483 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.275435 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-kube-api-access-4z2nl" pod "39235b85-adb6-477d-b069-64e29e05fd17" (UID: "39235b85-adb6-477d-b069-64e29e05fd17"). InnerVolumeSpecName "kube-api-access-4z2nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:05.276064 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.276038 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-istio-token" pod "39235b85-adb6-477d-b069-64e29e05fd17" (UID: "39235b85-adb6-477d-b069-64e29e05fd17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:05.276134 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.276107 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-kube-api-access-4jzvp" pod "b202666a-2b81-4d03-b286-a05d7a87b373" (UID: "b202666a-2b81-4d03-b286-a05d7a87b373"). InnerVolumeSpecName "kube-api-access-4jzvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:05.276601 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.276580 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-istio-token" pod "b202666a-2b81-4d03-b286-a05d7a87b373" (UID: "b202666a-2b81-4d03-b286-a05d7a87b373"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:05.277367 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.277348 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39235b85-adb6-477d-b069-64e29e05fd17-istio-podinfo" pod "39235b85-adb6-477d-b069-64e29e05fd17" (UID: "39235b85-adb6-477d-b069-64e29e05fd17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:05.374103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374074 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jzvp\" (UniqueName: \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-kube-api-access-4jzvp\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374104 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39235b85-adb6-477d-b069-64e29e05fd17-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374119 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374132 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374145 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374157 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374169 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b202666a-2b81-4d03-b286-a05d7a87b373-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374181 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2nl\" (UniqueName: \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-kube-api-access-4z2nl\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374192 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b202666a-2b81-4d03-b286-a05d7a87b373-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374203 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374214 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39235b85-adb6-477d-b069-64e29e05fd17-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374225 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b202666a-2b81-4d03-b286-a05d7a87b373-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.374314 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.374236 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39235b85-adb6-477d-b069-64e29e05fd17-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:05.538808 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.538708 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck"] Oct 30 14:26:05.539389 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539367 4968 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c05ff-d1c5-4e2a-80b7-143b4757e660" containerName="revoke-keys" Oct 30 14:26:05.539389 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539389 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c05ff-d1c5-4e2a-80b7-143b4757e660" containerName="revoke-keys" Oct 30 14:26:05.539558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539406 4968 cpu_manager.go:414] "containerMap: removing container" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" Oct 30 14:26:05.539558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539413 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" Oct 30 14:26:05.539558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539422 4968 cpu_manager.go:414] "containerMap: removing container" podUID="9d34ecd5-3570-487d-b86c-d315737475d5" containerName="revoke-keys" Oct 30 14:26:05.539558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539430 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34ecd5-3570-487d-b86c-d315737475d5" containerName="revoke-keys" Oct 30 14:26:05.539558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539474 4968 cpu_manager.go:414] "containerMap: removing container" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" Oct 30 14:26:05.539558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539481 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" Oct 30 14:26:05.539558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539503 4968 cpu_manager.go:414] "containerMap: removing container" podUID="39235b85-adb6-477d-b069-64e29e05fd17" containerName="istio-proxy" Oct 30 14:26:05.539558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539510 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39235b85-adb6-477d-b069-64e29e05fd17" containerName="istio-proxy" Oct 30 14:26:05.539976 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539674 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="39235b85-adb6-477d-b069-64e29e05fd17" containerName="istio-proxy" Oct 30 14:26:05.539976 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539694 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9c05ff-d1c5-4e2a-80b7-143b4757e660" containerName="revoke-keys" Oct 30 14:26:05.539976 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539708 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" Oct 30 14:26:05.539976 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539738 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34ecd5-3570-487d-b86c-d315737475d5" containerName="revoke-keys" Oct 30 14:26:05.539976 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.539753 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" containerName="istio-proxy" Oct 30 14:26:05.543499 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.543477 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:05.564399 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.564373 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck"] Oct 30 14:26:05.576524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.576494 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205590bb-txwck\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:05.577664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.576935 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205590bb-txwck\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:05.577664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.577006 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8777\" (UniqueName: \"kubernetes.io/projected/2732b5e3-df26-4e6c-927b-72edc4859022-kube-api-access-t8777\") pod \"maas-api-revoke-sub-205590bb-txwck\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:05.678373 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.678338 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205590bb-txwck\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:05.678567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.678396 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8777\" (UniqueName: \"kubernetes.io/projected/2732b5e3-df26-4e6c-927b-72edc4859022-kube-api-access-t8777\") pod \"maas-api-revoke-sub-205590bb-txwck\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:05.678567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.678459 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205590bb-txwck\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:05.679199 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.679171 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205590bb-txwck\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:05.680872 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.680849 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205590bb-txwck\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:05.700561 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.700528 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8777\" (UniqueName: \"kubernetes.io/projected/2732b5e3-df26-4e6c-927b-72edc4859022-kube-api-access-t8777\") pod \"maas-api-revoke-sub-205590bb-txwck\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:05.857233 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.857199 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:05.932521 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.928139 4968 generic.go:356] "Generic (PLEG): container finished" podID="39235b85-adb6-477d-b069-64e29e05fd17" containerID="2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c" exitCode=0 Oct 30 14:26:05.932521 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.928202 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" Oct 30 14:26:05.932521 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.928218 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" event={"ID":"39235b85-adb6-477d-b069-64e29e05fd17","Type":"ContainerDied","Data":"2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c"} Oct 30 14:26:05.932521 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.928289 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62" event={"ID":"39235b85-adb6-477d-b069-64e29e05fd17","Type":"ContainerDied","Data":"750ef2b9e6cc63d69e673ddfd608fcc8cb7248e1d812b44a61dda508a4fc69a1"} Oct 30 14:26:05.932521 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.928318 4968 scope.go:122] "RemoveContainer" containerID="2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c" Oct 30 14:26:05.932521 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.930758 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" event={"ID":"bcbbc581-0989-460d-b26d-2dc65b74974e","Type":"ContainerStarted","Data":"e15c2476ff44eb4123f11b592acb84d32c4cd70dc139f79ee9aaef8d360fcd6e"} Oct 30 14:26:05.932521 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.930800 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" event={"ID":"bcbbc581-0989-460d-b26d-2dc65b74974e","Type":"ContainerStarted","Data":"3451bccefbd5ff0e42bfc667ed2f107c74932677d5f4069ef63ba49ef2c70873"} Oct 30 14:26:05.937677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.933729 4968 generic.go:356] "Generic (PLEG): container finished" podID="b202666a-2b81-4d03-b286-a05d7a87b373" containerID="dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d" exitCode=0 Oct 30 14:26:05.937677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.933921 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" event={"ID":"b202666a-2b81-4d03-b286-a05d7a87b373","Type":"ContainerDied","Data":"dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d"} Oct 30 14:26:05.937677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.933958 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" event={"ID":"b202666a-2b81-4d03-b286-a05d7a87b373","Type":"ContainerDied","Data":"fe3299ba242f3c8cbd8a5710bd294f8e86f34b0aa18a19b29f7ffb046b94bbc2"} Oct 30 14:26:05.937677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.934042 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s" Oct 30 14:26:05.962616 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.962570 4968 scope.go:122] "RemoveContainer" containerID="2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c" Oct 30 14:26:05.962949 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:05.962920 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c\": container with ID starting with 2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c not found: ID does not exist" containerID="2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c" Oct 30 14:26:05.963048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.962959 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c"} err="failed to get container status \"2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c\": rpc error: code = NotFound desc = could not find container \"2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c\": container with ID starting with 2538b553d26b1709224bfbb25b156172d01b3b3c120581799d710e4c21c7681c not found: ID does not exist" Oct 30 14:26:05.963048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.962983 4968 scope.go:122] "RemoveContainer" containerID="dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d" Oct 30 14:26:05.982086 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.981985 4968 scope.go:122] "RemoveContainer" containerID="24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9" Oct 30 14:26:05.994889 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.994828 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" podStartSLOduration=1.994807607 podStartE2EDuration="1.994807607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:05.98460399 +0000 UTC m=+1161.610047363" watchObservedRunningTime="2026-10-30 14:26:05.994807607 +0000 UTC m=+1161.620250981" Oct 30 14:26:05.997379 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.997361 4968 scope.go:122] "RemoveContainer" containerID="dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d" Oct 30 14:26:05.997691 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:05.997668 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d\": container with ID starting with dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d not found: ID does not exist" containerID="dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d" Oct 30 14:26:05.997808 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.997698 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d"} err="failed to get container status \"dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d\": rpc error: code = NotFound desc = could not find container \"dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d\": container with ID starting with dac2bd06e6333d713bdf957eba0bc0a72dc084a9ff3840fa77fa96ff2d80042d not found: ID does not exist" Oct 30 14:26:05.997808 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.997732 4968 scope.go:122] "RemoveContainer" containerID="24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9" Oct 30 14:26:05.998061 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:05.998037 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9\": container with ID starting with 24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9 not found: ID does not exist" containerID="24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9" Oct 30 14:26:05.998116 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:05.998069 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9"} err="failed to get container status \"24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9\": rpc error: code = NotFound desc = could not find container \"24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9\": container with ID starting with 24051138a9c8409302bdbff01c612748ec55aefb98ab0b9daa56d36fe42ef3a9 not found: ID does not exist" Oct 30 14:26:06.025090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.025062 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62"] Oct 30 14:26:06.032479 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.032453 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-q7s62"] Oct 30 14:26:06.074307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.074275 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" podUID="9e1775f1-3d4b-491c-b2dd-f207e5bd8025" containerName="payload-processing" containerID="cri-o://2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4" gracePeriod=25 Oct 30 14:26:06.074837 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.074819 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s"] Oct 30 14:26:06.079152 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.079128 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a5ddf3-openshift-default-5fdbbf8547-wxc8s"] Oct 30 14:26:06.171697 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.171669 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck"] Oct 30 14:26:06.171963 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:26:06.171935 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2732b5e3_df26_4e6c_927b_72edc4859022.slice/crio-3a05ca0db8e05a012d02205f49f65ca902188efb49bb5bfda646b9861d89378f WatchSource:0}: Error finding container 3a05ca0db8e05a012d02205f49f65ca902188efb49bb5bfda646b9861d89378f: Status 404 returned error can't find the container with id 3a05ca0db8e05a012d02205f49f65ca902188efb49bb5bfda646b9861d89378f Oct 30 14:26:06.519528 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.519503 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39235b85-adb6-477d-b069-64e29e05fd17" path="/var/lib/kubelet/pods/39235b85-adb6-477d-b069-64e29e05fd17/volumes" Oct 30 14:26:06.519998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.519984 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b202666a-2b81-4d03-b286-a05d7a87b373" path="/var/lib/kubelet/pods/b202666a-2b81-4d03-b286-a05d7a87b373/volumes" Oct 30 14:26:06.943378 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.943343 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" event={"ID":"2732b5e3-df26-4e6c-927b-72edc4859022","Type":"ContainerStarted","Data":"2bc1c0509a6bcef16c7291765403ad1fc3eb9bcb6dcd3a6bbae367e4176b6d66"} Oct 30 14:26:06.943593 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.943397 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" event={"ID":"2732b5e3-df26-4e6c-927b-72edc4859022","Type":"ContainerStarted","Data":"3a05ca0db8e05a012d02205f49f65ca902188efb49bb5bfda646b9861d89378f"} Oct 30 14:26:06.946235 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.946205 4968 generic.go:356] "Generic (PLEG): container finished" podID="bcbbc581-0989-460d-b26d-2dc65b74974e" containerID="e15c2476ff44eb4123f11b592acb84d32c4cd70dc139f79ee9aaef8d360fcd6e" exitCode=0 Oct 30 14:26:06.946353 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.946241 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" event={"ID":"bcbbc581-0989-460d-b26d-2dc65b74974e","Type":"ContainerDied","Data":"e15c2476ff44eb4123f11b592acb84d32c4cd70dc139f79ee9aaef8d360fcd6e"} Oct 30 14:26:06.982905 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:06.982852 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" podStartSLOduration=1.98283493 podStartE2EDuration="1.98283493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:06.979575385 +0000 UTC m=+1162.605018759" watchObservedRunningTime="2026-10-30 14:26:06.98283493 +0000 UTC m=+1162.608278304" Oct 30 14:26:07.242293 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:07.242191 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" podUID="13dedc53-2e29-4eae-9ba3-267ae50634b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:26:07.242293 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:07.242213 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" podUID="9e1775f1-3d4b-491c-b2dd-f207e5bd8025" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:26:07.968990 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:07.968948 4968 generic.go:356] "Generic (PLEG): container finished" podID="2732b5e3-df26-4e6c-927b-72edc4859022" containerID="2bc1c0509a6bcef16c7291765403ad1fc3eb9bcb6dcd3a6bbae367e4176b6d66" exitCode=0 Oct 30 14:26:07.970493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:07.969776 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" event={"ID":"2732b5e3-df26-4e6c-927b-72edc4859022","Type":"ContainerDied","Data":"2bc1c0509a6bcef16c7291765403ad1fc3eb9bcb6dcd3a6bbae367e4176b6d66"} Oct 30 14:26:08.174117 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.174091 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:08.233471 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.233384 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcbbc581-0989-460d-b26d-2dc65b74974e-kube-api-access-79qvv\" (UniqueName: \"kubernetes.io/projected/bcbbc581-0989-460d-b26d-2dc65b74974e-kube-api-access-79qvv\") pod \"bcbbc581-0989-460d-b26d-2dc65b74974e\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " Oct 30 14:26:08.233652 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.233512 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-openshift-ca\") pod \"bcbbc581-0989-460d-b26d-2dc65b74974e\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " Oct 30 14:26:08.233652 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.233579 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-xks-ca\") pod \"bcbbc581-0989-460d-b26d-2dc65b74974e\" (UID: \"bcbbc581-0989-460d-b26d-2dc65b74974e\") " Oct 30 14:26:08.234763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.234499 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-openshift-ca" pod "bcbbc581-0989-460d-b26d-2dc65b74974e" (UID: "bcbbc581-0989-460d-b26d-2dc65b74974e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:08.236416 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.236379 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-xks-ca" pod "bcbbc581-0989-460d-b26d-2dc65b74974e" (UID: "bcbbc581-0989-460d-b26d-2dc65b74974e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:26:08.236556 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.236529 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbbc581-0989-460d-b26d-2dc65b74974e-kube-api-access-79qvv" pod "bcbbc581-0989-460d-b26d-2dc65b74974e" (UID: "bcbbc581-0989-460d-b26d-2dc65b74974e"). InnerVolumeSpecName "kube-api-access-79qvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:08.335152 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.335122 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:08.335152 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.335148 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79qvv\" (UniqueName: \"kubernetes.io/projected/bcbbc581-0989-460d-b26d-2dc65b74974e-kube-api-access-79qvv\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:08.335152 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.335157 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbbc581-0989-460d-b26d-2dc65b74974e-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:08.979634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.979588 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" Oct 30 14:26:08.979634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.979638 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9" event={"ID":"bcbbc581-0989-460d-b26d-2dc65b74974e","Type":"ContainerDied","Data":"3451bccefbd5ff0e42bfc667ed2f107c74932677d5f4069ef63ba49ef2c70873"} Oct 30 14:26:08.980248 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:08.979690 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3451bccefbd5ff0e42bfc667ed2f107c74932677d5f4069ef63ba49ef2c70873" Oct 30 14:26:09.163173 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.163149 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:09.246488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.246410 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2732b5e3-df26-4e6c-927b-72edc4859022-kube-api-access-t8777\" (UniqueName: \"kubernetes.io/projected/2732b5e3-df26-4e6c-927b-72edc4859022-kube-api-access-t8777\") pod \"2732b5e3-df26-4e6c-927b-72edc4859022\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " Oct 30 14:26:09.246646 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.246495 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-xks-ca\") pod \"2732b5e3-df26-4e6c-927b-72edc4859022\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " Oct 30 14:26:09.246646 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.246581 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-openshift-ca\") pod \"2732b5e3-df26-4e6c-927b-72edc4859022\" (UID: \"2732b5e3-df26-4e6c-927b-72edc4859022\") " Oct 30 14:26:09.247328 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.247294 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-openshift-ca" pod "2732b5e3-df26-4e6c-927b-72edc4859022" (UID: "2732b5e3-df26-4e6c-927b-72edc4859022"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:09.248803 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.248775 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-xks-ca" pod "2732b5e3-df26-4e6c-927b-72edc4859022" (UID: "2732b5e3-df26-4e6c-927b-72edc4859022"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:26:09.249195 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.249180 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2732b5e3-df26-4e6c-927b-72edc4859022-kube-api-access-t8777" pod "2732b5e3-df26-4e6c-927b-72edc4859022" (UID: "2732b5e3-df26-4e6c-927b-72edc4859022"). InnerVolumeSpecName "kube-api-access-t8777". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:09.347957 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.347919 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8777\" (UniqueName: \"kubernetes.io/projected/2732b5e3-df26-4e6c-927b-72edc4859022-kube-api-access-t8777\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:09.347957 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.347950 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:09.347957 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.347960 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2732b5e3-df26-4e6c-927b-72edc4859022-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:09.991114 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.991070 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" Oct 30 14:26:09.991114 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.991108 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck" event={"ID":"2732b5e3-df26-4e6c-927b-72edc4859022","Type":"ContainerDied","Data":"3a05ca0db8e05a012d02205f49f65ca902188efb49bb5bfda646b9861d89378f"} Oct 30 14:26:09.992237 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:09.991156 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a05ca0db8e05a012d02205f49f65ca902188efb49bb5bfda646b9861d89378f" Oct 30 14:26:10.264546 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.264463 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" podUID="b3ea3629-deb5-427f-849b-2e40f1bbabc5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:26:10.438611 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.438573 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" podUID="819c0707-1f66-400e-8704-c902586cae50" containerName="main" containerID="cri-o://db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3" gracePeriod=45 Oct 30 14:26:10.448190 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.448150 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" podUID="58305994-dcf4-44a4-afeb-b3e89cad85f1" containerName="main" containerID="cri-o://584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9" gracePeriod=45 Oct 30 14:26:10.759250 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.759223 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:26:10.762580 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.762557 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:26:10.865108 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865080 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-dshm\") pod \"58305994-dcf4-44a4-afeb-b3e89cad85f1\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " Oct 30 14:26:10.865266 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865169 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-tmp-dir\") pod \"819c0707-1f66-400e-8704-c902586cae50\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " Oct 30 14:26:10.865266 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865213 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-model-cache\") pod \"819c0707-1f66-400e-8704-c902586cae50\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " Oct 30 14:26:10.865395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865263 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-tmp-dir\") pod \"58305994-dcf4-44a4-afeb-b3e89cad85f1\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " Oct 30 14:26:10.865395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865296 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-home\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-home\") pod \"819c0707-1f66-400e-8704-c902586cae50\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " Oct 30 14:26:10.865395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865326 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-dshm\") pod \"819c0707-1f66-400e-8704-c902586cae50\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " Oct 30 14:26:10.865395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865378 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58305994-dcf4-44a4-afeb-b3e89cad85f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/58305994-dcf4-44a4-afeb-b3e89cad85f1-tls-certs\") pod \"58305994-dcf4-44a4-afeb-b3e89cad85f1\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " Oct 30 14:26:10.865603 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865452 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819c0707-1f66-400e-8704-c902586cae50-kube-api-access-xhz5c\" (UniqueName: \"kubernetes.io/projected/819c0707-1f66-400e-8704-c902586cae50-kube-api-access-xhz5c\") pod \"819c0707-1f66-400e-8704-c902586cae50\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " Oct 30 14:26:10.865603 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865464 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-tmp-dir" pod "819c0707-1f66-400e-8704-c902586cae50" (UID: "819c0707-1f66-400e-8704-c902586cae50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:10.865603 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865515 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-model-cache" pod "819c0707-1f66-400e-8704-c902586cae50" (UID: "819c0707-1f66-400e-8704-c902586cae50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:10.865603 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865523 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-model-cache\") pod \"58305994-dcf4-44a4-afeb-b3e89cad85f1\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " Oct 30 14:26:10.865603 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865591 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-home\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-home\") pod \"58305994-dcf4-44a4-afeb-b3e89cad85f1\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " Oct 30 14:26:10.865968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865653 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/819c0707-1f66-400e-8704-c902586cae50-tls-certs\" (UniqueName: \"kubernetes.io/secret/819c0707-1f66-400e-8704-c902586cae50-tls-certs\") pod \"819c0707-1f66-400e-8704-c902586cae50\" (UID: \"819c0707-1f66-400e-8704-c902586cae50\") " Oct 30 14:26:10.865968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865689 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-home" pod "819c0707-1f66-400e-8704-c902586cae50" (UID: "819c0707-1f66-400e-8704-c902586cae50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:10.865968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865698 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-model-cache" pod "58305994-dcf4-44a4-afeb-b3e89cad85f1" (UID: "58305994-dcf4-44a4-afeb-b3e89cad85f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:10.865968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865769 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58305994-dcf4-44a4-afeb-b3e89cad85f1-kube-api-access-slwn9\" (UniqueName: \"kubernetes.io/projected/58305994-dcf4-44a4-afeb-b3e89cad85f1-kube-api-access-slwn9\") pod \"58305994-dcf4-44a4-afeb-b3e89cad85f1\" (UID: \"58305994-dcf4-44a4-afeb-b3e89cad85f1\") " Oct 30 14:26:10.865968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.865930 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-tmp-dir" pod "58305994-dcf4-44a4-afeb-b3e89cad85f1" (UID: "58305994-dcf4-44a4-afeb-b3e89cad85f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:10.866349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.866285 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:10.866557 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.866526 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-home" pod "58305994-dcf4-44a4-afeb-b3e89cad85f1" (UID: "58305994-dcf4-44a4-afeb-b3e89cad85f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:10.866768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.866311 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:10.866768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.866724 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:10.866768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.866739 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:10.866768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.866750 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:10.873731 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.868766 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58305994-dcf4-44a4-afeb-b3e89cad85f1-tls-certs" pod "58305994-dcf4-44a4-afeb-b3e89cad85f1" (UID: "58305994-dcf4-44a4-afeb-b3e89cad85f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:26:10.873731 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.869091 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58305994-dcf4-44a4-afeb-b3e89cad85f1-kube-api-access-slwn9" pod "58305994-dcf4-44a4-afeb-b3e89cad85f1" (UID: "58305994-dcf4-44a4-afeb-b3e89cad85f1"). InnerVolumeSpecName "kube-api-access-slwn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:10.879927 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.879900 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-dshm" pod "58305994-dcf4-44a4-afeb-b3e89cad85f1" (UID: "58305994-dcf4-44a4-afeb-b3e89cad85f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:10.884996 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.884938 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-dshm" pod "819c0707-1f66-400e-8704-c902586cae50" (UID: "819c0707-1f66-400e-8704-c902586cae50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:10.884996 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.884957 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819c0707-1f66-400e-8704-c902586cae50-tls-certs" pod "819c0707-1f66-400e-8704-c902586cae50" (UID: "819c0707-1f66-400e-8704-c902586cae50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:26:10.885135 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.884981 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819c0707-1f66-400e-8704-c902586cae50-kube-api-access-xhz5c" pod "819c0707-1f66-400e-8704-c902586cae50" (UID: "819c0707-1f66-400e-8704-c902586cae50"). InnerVolumeSpecName "kube-api-access-xhz5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:10.968466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.968430 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819c0707-1f66-400e-8704-c902586cae50-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:10.968586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.968495 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58305994-dcf4-44a4-afeb-b3e89cad85f1-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:10.968586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.968514 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhz5c\" (UniqueName: \"kubernetes.io/projected/819c0707-1f66-400e-8704-c902586cae50-kube-api-access-xhz5c\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:10.968586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.968527 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:10.968586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.968540 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819c0707-1f66-400e-8704-c902586cae50-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:10.968586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.968552 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slwn9\" (UniqueName: \"kubernetes.io/projected/58305994-dcf4-44a4-afeb-b3e89cad85f1-kube-api-access-slwn9\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:10.968586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:10.968563 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58305994-dcf4-44a4-afeb-b3e89cad85f1-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:11.027477 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.022027 4968 generic.go:356] "Generic (PLEG): container finished" podID="58305994-dcf4-44a4-afeb-b3e89cad85f1" containerID="584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9" exitCode=0 Oct 30 14:26:11.027477 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.022115 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" event={"ID":"58305994-dcf4-44a4-afeb-b3e89cad85f1","Type":"ContainerDied","Data":"584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9"} Oct 30 14:26:11.027477 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.022152 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" event={"ID":"58305994-dcf4-44a4-afeb-b3e89cad85f1","Type":"ContainerDied","Data":"515a627fdb47e4921756c229a9dca9707e1c49eab489ffcc8d8e9561d9310ce1"} Oct 30 14:26:11.027477 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.022173 4968 scope.go:122] "RemoveContainer" containerID="584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9" Oct 30 14:26:11.027477 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.022392 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd" Oct 30 14:26:11.037516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.037484 4968 generic.go:356] "Generic (PLEG): container finished" podID="819c0707-1f66-400e-8704-c902586cae50" containerID="db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3" exitCode=0 Oct 30 14:26:11.037897 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.037873 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" event={"ID":"819c0707-1f66-400e-8704-c902586cae50","Type":"ContainerDied","Data":"db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3"} Oct 30 14:26:11.037993 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.037915 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" event={"ID":"819c0707-1f66-400e-8704-c902586cae50","Type":"ContainerDied","Data":"b56300d270b2f40120197e9f185a4dc1d9b90b65ad95f44f52991918409500ae"} Oct 30 14:26:11.038049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.037992 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8" Oct 30 14:26:11.041821 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.041793 4968 generic.go:356] "Generic (PLEG): container finished" podID="13dedc53-2e29-4eae-9ba3-267ae50634b4" containerID="e83d84ec2ca8891666231b06789e9a3a7599ae9e95bd2ac0163c076201d6f979" exitCode=0 Oct 30 14:26:11.042396 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.042367 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" event={"ID":"13dedc53-2e29-4eae-9ba3-267ae50634b4","Type":"ContainerDied","Data":"e83d84ec2ca8891666231b06789e9a3a7599ae9e95bd2ac0163c076201d6f979"} Oct 30 14:26:11.051525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.051372 4968 scope.go:122] "RemoveContainer" containerID="584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9" Oct 30 14:26:11.051843 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:11.051817 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9\": container with ID starting with 584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9 not found: ID does not exist" containerID="584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9" Oct 30 14:26:11.051954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.051856 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9"} err="failed to get container status \"584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9\": rpc error: code = NotFound desc = could not find container \"584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9\": container with ID starting with 584901c8f712b6816207656d40be88213f4c96feef8b07dd9c84dbca3f16bed9 not found: ID does not exist" Oct 30 14:26:11.051954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.051888 4968 scope.go:122] "RemoveContainer" containerID="db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3" Oct 30 14:26:11.064451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.064420 4968 scope.go:122] "RemoveContainer" containerID="db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3" Oct 30 14:26:11.064788 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:11.064767 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3\": container with ID starting with db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3 not found: ID does not exist" containerID="db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3" Oct 30 14:26:11.064859 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.064797 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3"} err="failed to get container status \"db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3\": rpc error: code = NotFound desc = could not find container \"db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3\": container with ID starting with db6c48e954521a63b462debd3dfe393da00edc7f475162f248f492b637acb9c3 not found: ID does not exist" Oct 30 14:26:11.093737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.092760 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd"] Oct 30 14:26:11.102806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.101466 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-w6-bf8266-kserve-7b95b4754f-zdqmd"] Oct 30 14:26:11.130805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.130772 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8"] Oct 30 14:26:11.135408 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.135381 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-bf8266/distinct-2-w6-bf8266-kserve-66f5bc94f4-xrzs8"] Oct 30 14:26:11.149269 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.149247 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:26:11.276351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.274984 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13dedc53-2e29-4eae-9ba3-267ae50634b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dedc53-2e29-4eae-9ba3-267ae50634b4-plugins-config-volume\") pod \"13dedc53-2e29-4eae-9ba3-267ae50634b4\" (UID: \"13dedc53-2e29-4eae-9ba3-267ae50634b4\") " Oct 30 14:26:11.276351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.275045 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13dedc53-2e29-4eae-9ba3-267ae50634b4-kube-api-access-bhv4b\" (UniqueName: \"kubernetes.io/projected/13dedc53-2e29-4eae-9ba3-267ae50634b4-kube-api-access-bhv4b\") pod \"13dedc53-2e29-4eae-9ba3-267ae50634b4\" (UID: \"13dedc53-2e29-4eae-9ba3-267ae50634b4\") " Oct 30 14:26:11.277431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.277369 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13dedc53-2e29-4eae-9ba3-267ae50634b4-plugins-config-volume" pod "13dedc53-2e29-4eae-9ba3-267ae50634b4" (UID: "13dedc53-2e29-4eae-9ba3-267ae50634b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:11.326740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.318911 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13dedc53-2e29-4eae-9ba3-267ae50634b4-kube-api-access-bhv4b" pod "13dedc53-2e29-4eae-9ba3-267ae50634b4" (UID: "13dedc53-2e29-4eae-9ba3-267ae50634b4"). InnerVolumeSpecName "kube-api-access-bhv4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:11.394795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.385645 4968 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dedc53-2e29-4eae-9ba3-267ae50634b4-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:11.394795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.385686 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhv4b\" (UniqueName: \"kubernetes.io/projected/13dedc53-2e29-4eae-9ba3-267ae50634b4-kube-api-access-bhv4b\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:11.720258 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.720165 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p"] Oct 30 14:26:11.724576 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:11.724548 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-24dfd7-xl99p"] Oct 30 14:26:12.052207 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:12.052122 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" Oct 30 14:26:12.052207 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:12.052122 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw" event={"ID":"13dedc53-2e29-4eae-9ba3-267ae50634b4","Type":"ContainerDied","Data":"9487136c04841be2d7292183fb04e9dfae881c89ecf638ab0707a1dee7d9d59b"} Oct 30 14:26:12.052829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:12.052252 4968 scope.go:122] "RemoveContainer" containerID="e83d84ec2ca8891666231b06789e9a3a7599ae9e95bd2ac0163c076201d6f979" Oct 30 14:26:12.101780 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:12.101737 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw"] Oct 30 14:26:12.111694 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:12.111670 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-24dfd7-5cdbbb87bf-4xpnw"] Oct 30 14:26:12.274462 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:12.274426 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" podUID="0686340e-12ce-45b1-8135-2cf61f0a3f0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:26:12.521915 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:12.521875 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13dedc53-2e29-4eae-9ba3-267ae50634b4" path="/var/lib/kubelet/pods/13dedc53-2e29-4eae-9ba3-267ae50634b4/volumes" Oct 30 14:26:12.522438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:12.522416 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c" path="/var/lib/kubelet/pods/38bc4c5d-0a42-4842-aba5-4ba1d15aeb2c/volumes" Oct 30 14:26:12.522904 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:12.522885 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58305994-dcf4-44a4-afeb-b3e89cad85f1" path="/var/lib/kubelet/pods/58305994-dcf4-44a4-afeb-b3e89cad85f1/volumes" Oct 30 14:26:12.523632 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:12.523612 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819c0707-1f66-400e-8704-c902586cae50" path="/var/lib/kubelet/pods/819c0707-1f66-400e-8704-c902586cae50/volumes" Oct 30 14:26:14.328741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:14.327769 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64"] Oct 30 14:26:14.352754 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:14.334228 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1d36cf-fhc64"] Oct 30 14:26:14.755914 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:14.753969 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1b5298-d140-48d3-830c-ac7c2ec4ef14" path="/var/lib/kubelet/pods/1b1b5298-d140-48d3-830c-ac7c2ec4ef14/volumes" Oct 30 14:26:15.067877 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.067778 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq"] Oct 30 14:26:15.071749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.069377 4968 cpu_manager.go:414] "containerMap: removing container" podUID="2732b5e3-df26-4e6c-927b-72edc4859022" containerName="revoke-keys" Oct 30 14:26:15.071929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.071908 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2732b5e3-df26-4e6c-927b-72edc4859022" containerName="revoke-keys" Oct 30 14:26:15.072059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.072046 4968 cpu_manager.go:414] "containerMap: removing container" podUID="819c0707-1f66-400e-8704-c902586cae50" containerName="main" Oct 30 14:26:15.072151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.072138 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819c0707-1f66-400e-8704-c902586cae50" containerName="main" Oct 30 14:26:15.072354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.072337 4968 cpu_manager.go:414] "containerMap: removing container" podUID="bcbbc581-0989-460d-b26d-2dc65b74974e" containerName="revoke-keys" Oct 30 14:26:15.072466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.072453 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbbc581-0989-460d-b26d-2dc65b74974e" containerName="revoke-keys" Oct 30 14:26:15.072582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.072571 4968 cpu_manager.go:414] "containerMap: removing container" podUID="13dedc53-2e29-4eae-9ba3-267ae50634b4" containerName="payload-processing" Oct 30 14:26:15.074811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.074791 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13dedc53-2e29-4eae-9ba3-267ae50634b4" containerName="payload-processing" Oct 30 14:26:15.074961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.074947 4968 cpu_manager.go:414] "containerMap: removing container" podUID="58305994-dcf4-44a4-afeb-b3e89cad85f1" containerName="main" Oct 30 14:26:15.075052 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.075040 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58305994-dcf4-44a4-afeb-b3e89cad85f1" containerName="main" Oct 30 14:26:15.075453 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.075434 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="2732b5e3-df26-4e6c-927b-72edc4859022" containerName="revoke-keys" Oct 30 14:26:15.075575 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.075562 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="819c0707-1f66-400e-8704-c902586cae50" containerName="main" Oct 30 14:26:15.075689 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.075676 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbbc581-0989-460d-b26d-2dc65b74974e" containerName="revoke-keys" Oct 30 14:26:15.079036 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.079021 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="13dedc53-2e29-4eae-9ba3-267ae50634b4" containerName="payload-processing" Oct 30 14:26:15.079154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.079144 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="58305994-dcf4-44a4-afeb-b3e89cad85f1" containerName="main" Oct 30 14:26:15.239082 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.238952 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq"] Oct 30 14:26:15.239442 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.239418 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.282748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.282069 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7bf15f\"/\"distinct-w5-7bf15f-kserve-self-signed-certs\"" Oct 30 14:26:15.360576 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.360333 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-dshm\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.361102 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.361076 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tls-certs\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.361465 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.361445 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx2fs\" (UniqueName: \"kubernetes.io/projected/d9555cbc-5986-436f-8d2e-4550cbe5e49c-kube-api-access-jx2fs\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.361759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.361705 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-model-cache\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.362242 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.362219 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-home\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.362453 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.362435 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tmp-dir\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.465739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.464418 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tmp-dir\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.465739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.464503 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-dshm\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.465739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.464543 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tls-certs\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.465739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.464580 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2fs\" (UniqueName: \"kubernetes.io/projected/d9555cbc-5986-436f-8d2e-4550cbe5e49c-kube-api-access-jx2fs\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.465739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.464739 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-model-cache\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.465739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.464787 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-home\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.465739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.465271 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-home\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.465739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.465523 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tmp-dir\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.466892 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.466861 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-model-cache\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.471493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.471464 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-dshm\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.472030 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.472008 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tls-certs\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.491918 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.491581 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2fs\" (UniqueName: \"kubernetes.io/projected/d9555cbc-5986-436f-8d2e-4550cbe5e49c-kube-api-access-jx2fs\") pod \"distinct-w5-7bf15f-kserve-7cfb889d7-zbznq\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:15.677063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:15.676963 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:16.021084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:16.021060 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq"] Oct 30 14:26:16.024762 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:26:16.024701 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9555cbc_5986_436f_8d2e_4550cbe5e49c.slice/crio-7a5f23c906513261c05447d54a459a1541525f0ca93cc512a5ac8d244492c42a WatchSource:0}: Error finding container 7a5f23c906513261c05447d54a459a1541525f0ca93cc512a5ac8d244492c42a: Status 404 returned error can't find the container with id 7a5f23c906513261c05447d54a459a1541525f0ca93cc512a5ac8d244492c42a Oct 30 14:26:16.211188 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:16.211154 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" event={"ID":"d9555cbc-5986-436f-8d2e-4550cbe5e49c","Type":"ContainerStarted","Data":"47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4"} Oct 30 14:26:16.211361 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:16.211202 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" event={"ID":"d9555cbc-5986-436f-8d2e-4550cbe5e49c","Type":"ContainerStarted","Data":"7a5f23c906513261c05447d54a459a1541525f0ca93cc512a5ac8d244492c42a"} Oct 30 14:26:16.266335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:16.266286 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" podStartSLOduration=2.266273167 podStartE2EDuration="2.266273167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:16.263215015 +0000 UTC m=+1171.888658387" watchObservedRunningTime="2026-10-30 14:26:16.266273167 +0000 UTC m=+1171.891716540" Oct 30 14:26:16.677396 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:16.677368 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:17.252007 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.251964 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" podUID="9e1775f1-3d4b-491c-b2dd-f207e5bd8025" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:26:17.272732 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.272277 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt"] Oct 30 14:26:17.277171 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.277149 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.311365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.309751 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-credential-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.311365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.309802 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-token\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.311365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.309870 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-podinfo\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.311365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.309912 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-data\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.311365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.310032 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-envoy\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.311365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.310098 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl8vb\" (UniqueName: \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-kube-api-access-rl8vb\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.311365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.310142 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istiod-ca-cert\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.311365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.310181 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.311365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.310209 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-certs\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.314405 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.313898 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt"] Oct 30 14:26:17.411610 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.411563 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl8vb\" (UniqueName: \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-kube-api-access-rl8vb\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.411828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.411639 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istiod-ca-cert\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.411828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.411684 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.411828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.411740 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-certs\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.412018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.411831 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-credential-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.412018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.411865 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-token\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.412018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.411936 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-podinfo\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.412018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.411978 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-data\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.412295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.412092 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-envoy\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.414980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.412859 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-certs\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.414980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.413833 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istiod-ca-cert\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.414980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.414103 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.414980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.414585 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-credential-socket\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.419799 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.419751 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-data\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.429049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.429021 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-envoy\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.431801 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.431779 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-podinfo\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.437632 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.437608 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-token\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.445813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.445789 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl8vb\" (UniqueName: \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-kube-api-access-rl8vb\") pod \"e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.475130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.475089 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:26:17.512572 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.512535 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ea3629-deb5-427f-849b-2e40f1bbabc5-kube-api-access-6jqgh\" (UniqueName: \"kubernetes.io/projected/b3ea3629-deb5-427f-849b-2e40f1bbabc5-kube-api-access-6jqgh\") pod \"b3ea3629-deb5-427f-849b-2e40f1bbabc5\" (UID: \"b3ea3629-deb5-427f-849b-2e40f1bbabc5\") " Oct 30 14:26:17.512757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.512611 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ea3629-deb5-427f-849b-2e40f1bbabc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ea3629-deb5-427f-849b-2e40f1bbabc5-plugins-config-volume\") pod \"b3ea3629-deb5-427f-849b-2e40f1bbabc5\" (UID: \"b3ea3629-deb5-427f-849b-2e40f1bbabc5\") " Oct 30 14:26:17.513745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.513678 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ea3629-deb5-427f-849b-2e40f1bbabc5-plugins-config-volume" pod "b3ea3629-deb5-427f-849b-2e40f1bbabc5" (UID: "b3ea3629-deb5-427f-849b-2e40f1bbabc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:17.515212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.515185 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ea3629-deb5-427f-849b-2e40f1bbabc5-kube-api-access-6jqgh" pod "b3ea3629-deb5-427f-849b-2e40f1bbabc5" (UID: "b3ea3629-deb5-427f-849b-2e40f1bbabc5"). InnerVolumeSpecName "kube-api-access-6jqgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:17.545989 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.545961 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:26:17.614444 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.614353 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0686340e-12ce-45b1-8135-2cf61f0a3f0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0686340e-12ce-45b1-8135-2cf61f0a3f0b-plugins-config-volume\") pod \"0686340e-12ce-45b1-8135-2cf61f0a3f0b\" (UID: \"0686340e-12ce-45b1-8135-2cf61f0a3f0b\") " Oct 30 14:26:17.614444 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.614406 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0686340e-12ce-45b1-8135-2cf61f0a3f0b-kube-api-access-h248c\" (UniqueName: \"kubernetes.io/projected/0686340e-12ce-45b1-8135-2cf61f0a3f0b-kube-api-access-h248c\") pod \"0686340e-12ce-45b1-8135-2cf61f0a3f0b\" (UID: \"0686340e-12ce-45b1-8135-2cf61f0a3f0b\") " Oct 30 14:26:17.615074 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.615048 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0686340e-12ce-45b1-8135-2cf61f0a3f0b-plugins-config-volume" pod "0686340e-12ce-45b1-8135-2cf61f0a3f0b" (UID: "0686340e-12ce-45b1-8135-2cf61f0a3f0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:17.615229 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.615139 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jqgh\" (UniqueName: \"kubernetes.io/projected/b3ea3629-deb5-427f-849b-2e40f1bbabc5-kube-api-access-6jqgh\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:17.615339 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.615324 4968 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ea3629-deb5-427f-849b-2e40f1bbabc5-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:17.617603 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.617572 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0686340e-12ce-45b1-8135-2cf61f0a3f0b-kube-api-access-h248c" pod "0686340e-12ce-45b1-8135-2cf61f0a3f0b" (UID: "0686340e-12ce-45b1-8135-2cf61f0a3f0b"). InnerVolumeSpecName "kube-api-access-h248c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:17.632518 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.632487 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:17.718149 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.716406 4968 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0686340e-12ce-45b1-8135-2cf61f0a3f0b-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:17.718149 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.716441 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h248c\" (UniqueName: \"kubernetes.io/projected/0686340e-12ce-45b1-8135-2cf61f0a3f0b-kube-api-access-h248c\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:17.983989 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:26:17.983944 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285f4c10_d262_43f5_a3b8_370e4bfe4d04.slice/crio-95ad0eef4f7772a9f401f1467aa80475e10fe55bc6154bd795caf118b8d7fc4c WatchSource:0}: Error finding container 95ad0eef4f7772a9f401f1467aa80475e10fe55bc6154bd795caf118b8d7fc4c: Status 404 returned error can't find the container with id 95ad0eef4f7772a9f401f1467aa80475e10fe55bc6154bd795caf118b8d7fc4c Oct 30 14:26:17.984134 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.984062 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt"] Oct 30 14:26:17.986932 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.986892 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:17.987026 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.986974 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:17.987026 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:17.987018 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:18.278798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.277741 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" event={"ID":"285f4c10-d262-43f5-a3b8-370e4bfe4d04","Type":"ContainerStarted","Data":"37d00805a50e7fa7a6f1fddf7621dfc0123671ef817feeee64b96b0c73aed900"} Oct 30 14:26:18.278798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.277845 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" event={"ID":"285f4c10-d262-43f5-a3b8-370e4bfe4d04","Type":"ContainerStarted","Data":"95ad0eef4f7772a9f401f1467aa80475e10fe55bc6154bd795caf118b8d7fc4c"} Oct 30 14:26:18.281330 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.281296 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" event={"ID":"b3ea3629-deb5-427f-849b-2e40f1bbabc5","Type":"ContainerDied","Data":"88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a"} Oct 30 14:26:18.281462 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.281409 4968 scope.go:122] "RemoveContainer" containerID="88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a" Oct 30 14:26:18.281462 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.281274 4968 generic.go:356] "Generic (PLEG): container finished" podID="b3ea3629-deb5-427f-849b-2e40f1bbabc5" containerID="88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a" exitCode=0 Oct 30 14:26:18.281588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.281534 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" event={"ID":"b3ea3629-deb5-427f-849b-2e40f1bbabc5","Type":"ContainerDied","Data":"33772b440407ed0777edc9e80b7dce0eb35b19b2feb08bed72d3c41f9fdf0c6c"} Oct 30 14:26:18.283981 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.283959 4968 generic.go:356] "Generic (PLEG): container finished" podID="0686340e-12ce-45b1-8135-2cf61f0a3f0b" containerID="a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666" exitCode=0 Oct 30 14:26:18.284829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.284542 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" Oct 30 14:26:18.284829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.284732 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" event={"ID":"0686340e-12ce-45b1-8135-2cf61f0a3f0b","Type":"ContainerDied","Data":"a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666"} Oct 30 14:26:18.284829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.284782 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28" event={"ID":"0686340e-12ce-45b1-8135-2cf61f0a3f0b","Type":"ContainerDied","Data":"ac620769ef8307cedede84a4befc5718bff910447bce652b8eaa9a96ef164b44"} Oct 30 14:26:18.311338 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.311316 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb" Oct 30 14:26:18.331569 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.331544 4968 scope.go:122] "RemoveContainer" containerID="88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a" Oct 30 14:26:18.332203 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:18.332180 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a\": container with ID starting with 88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a not found: ID does not exist" containerID="88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a" Oct 30 14:26:18.332369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.332344 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a"} err="failed to get container status \"88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a\": rpc error: code = NotFound desc = could not find container \"88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a\": container with ID starting with 88bf60cd160c646c13f5cd7b3f124be3b4035ebe469d316f5e3243c579d85a4a not found: ID does not exist" Oct 30 14:26:18.332470 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.332459 4968 scope.go:122] "RemoveContainer" containerID="a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666" Oct 30 14:26:18.334913 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.334830 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" podStartSLOduration=1.334817275 podStartE2EDuration="1.334817275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:18.331984294 +0000 UTC m=+1173.957427669" watchObservedRunningTime="2026-10-30 14:26:18.334817275 +0000 UTC m=+1173.960260647" Oct 30 14:26:18.348312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.348287 4968 scope.go:122] "RemoveContainer" containerID="a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666" Oct 30 14:26:18.348616 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:18.348595 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666\": container with ID starting with a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666 not found: ID does not exist" containerID="a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666" Oct 30 14:26:18.348680 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.348624 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666"} err="failed to get container status \"a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666\": rpc error: code = NotFound desc = could not find container \"a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666\": container with ID starting with a28595a6fad3ea6d16cc73e25f15022716e53edb77978746b77cf7e4c13bb666 not found: ID does not exist" Oct 30 14:26:18.375802 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.375773 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb"] Oct 30 14:26:18.381457 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.381432 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1d36cf-6f964584d7-sg4gb"] Oct 30 14:26:18.411515 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.410797 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28"] Oct 30 14:26:18.419563 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.419530 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1d36cf-dd6d47f75-hfn28"] Oct 30 14:26:18.523668 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.523634 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0686340e-12ce-45b1-8135-2cf61f0a3f0b" path="/var/lib/kubelet/pods/0686340e-12ce-45b1-8135-2cf61f0a3f0b/volumes" Oct 30 14:26:18.524344 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.524323 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ea3629-deb5-427f-849b-2e40f1bbabc5" path="/var/lib/kubelet/pods/b3ea3629-deb5-427f-849b-2e40f1bbabc5/volumes" Oct 30 14:26:18.633573 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:18.633539 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:19.395303 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:19.395268 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv"] Oct 30 14:26:19.427750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:19.417815 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bf8266-mwlxv"] Oct 30 14:26:19.635905 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:19.635821 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 14:26:19.635905 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:19.635888 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" podUID="285f4c10-d262-43f5-a3b8-370e4bfe4d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 14:26:20.448616 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:20.448572 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:26:20.537979 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:20.537943 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd844848-51b7-4c5c-9c4d-289622911ccc" path="/var/lib/kubelet/pods/dd844848-51b7-4c5c-9c4d-289622911ccc/volumes" Oct 30 14:26:20.633738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:20.633134 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 14:26:20.633738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:20.633183 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" podUID="285f4c10-d262-43f5-a3b8-370e4bfe4d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 14:26:21.673945 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:21.673869 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 14:26:21.674451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:21.673969 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" podUID="285f4c10-d262-43f5-a3b8-370e4bfe4d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 14:26:21.765005 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:21.764964 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh"] Oct 30 14:26:21.782737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:21.782654 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" containerID="cri-o://c706d0ae0e30a21cd85d4030607cb29d1872df9d7aa7ca9a834d91f36709d023" gracePeriod=30 Oct 30 14:26:21.898647 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:21.898607 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb"] Oct 30 14:26:21.945820 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:21.933177 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" containerID="cri-o://6411995df9ab33485e1e1dc0bfcee82487ff5a23a26c11ac02ca44bbfd95af72" gracePeriod=30 Oct 30 14:26:22.108930 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:22.108884 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc"] Oct 30 14:26:22.109481 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:22.109442 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" podUID="52388676-c0d2-4229-86ac-ad7763946b31" containerName="istio-proxy" containerID="cri-o://338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47" gracePeriod=30 Oct 30 14:26:22.661883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:22.661827 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 14:26:22.662220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:22.662181 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" podUID="285f4c10-d262-43f5-a3b8-370e4bfe4d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 14:26:22.861825 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:22.861775 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5"] Oct 30 14:26:22.862701 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:22.862646 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" containerID="cri-o://a4454be6f86b42b9449b6003a2a589fc6efc1c83ad13cce70b526079c2158bbd" gracePeriod=30 Oct 30 14:26:23.005800 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:22.989007 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt"] Oct 30 14:26:23.005800 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:22.989392 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" podUID="285f4c10-d262-43f5-a3b8-370e4bfe4d04" containerName="istio-proxy" containerID="cri-o://37d00805a50e7fa7a6f1fddf7621dfc0123671ef817feeee64b96b0c73aed900" gracePeriod=30 Oct 30 14:26:23.249534 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:23.249492 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf"] Oct 30 14:26:23.250164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:23.250114 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" containerID="cri-o://a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97" gracePeriod=30 Oct 30 14:26:23.446747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:23.445367 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:26:23.446747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:23.445426 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:26:23.969668 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:23.969633 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:26:23.970081 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:23.969688 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" podUID="52388676-c0d2-4229-86ac-ad7763946b31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:26:24.967459 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:24.967428 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5"] Oct 30 14:26:24.968013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:24.967998 4968 cpu_manager.go:414] "containerMap: removing container" podUID="0686340e-12ce-45b1-8135-2cf61f0a3f0b" containerName="payload-processing" Oct 30 14:26:24.968057 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:24.968013 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0686340e-12ce-45b1-8135-2cf61f0a3f0b" containerName="payload-processing" Oct 30 14:26:24.968104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:24.968058 4968 cpu_manager.go:414] "containerMap: removing container" podUID="b3ea3629-deb5-427f-849b-2e40f1bbabc5" containerName="payload-pre-processing" Oct 30 14:26:24.968104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:24.968063 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ea3629-deb5-427f-849b-2e40f1bbabc5" containerName="payload-pre-processing" Oct 30 14:26:24.968172 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:24.968139 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ea3629-deb5-427f-849b-2e40f1bbabc5" containerName="payload-pre-processing" Oct 30 14:26:24.968172 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:24.968150 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="0686340e-12ce-45b1-8135-2cf61f0a3f0b" containerName="payload-processing" Oct 30 14:26:24.973203 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:24.973184 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.014659 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.014621 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-137f01\"/\"distinct-w4-137f01-kserve-self-signed-certs\"" Oct 30 14:26:25.020543 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.020513 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5"] Oct 30 14:26:25.118453 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.118418 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5npzq\" (UniqueName: \"kubernetes.io/projected/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-kube-api-access-5npzq\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.118653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.118599 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tls-certs\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.118653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.118642 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tmp-dir\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.118777 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.118762 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-dshm\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.118834 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.118802 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-home\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.118939 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.118926 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-model-cache\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.151477 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.151411 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:26:25.151696 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.151671 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:26:25.224956 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.224855 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tls-certs\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.225460 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.225431 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tmp-dir\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.225616 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.225599 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-dshm\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.225764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.225748 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-home\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.226000 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.225982 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-model-cache\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.226163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.226148 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5npzq\" (UniqueName: \"kubernetes.io/projected/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-kube-api-access-5npzq\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.227394 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.227366 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-home\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.227671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.227645 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-model-cache\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.234781 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.227924 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tmp-dir\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.251518 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.251489 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5npzq\" (UniqueName: \"kubernetes.io/projected/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-kube-api-access-5npzq\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.264841 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.264801 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-dshm\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.266478 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.265333 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tls-certs\") pod \"distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.327621 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.321965 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:25.681337 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.679627 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:25.699228 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.699200 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5"] Oct 30 14:26:25.710792 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:25.710248 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:26.139796 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.139755 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" event={"ID":"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908","Type":"ContainerStarted","Data":"52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2"} Oct 30 14:26:26.140309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.139809 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" event={"ID":"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908","Type":"ContainerStarted","Data":"e575d1aecc6010daeb25087ecd6ba6abc58381a887a4f88eaad299e22b361a36"} Oct 30 14:26:26.150924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.150893 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:26:26.163167 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.163136 4968 patch_prober.go:29] interesting pod/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:26:26.163309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.163191 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:26:26.222706 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.222636 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" podStartSLOduration=2.222616917 podStartE2EDuration="2.222616917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:26.214674152 +0000 UTC m=+1181.840117546" watchObservedRunningTime="2026-10-30 14:26:26.222616917 +0000 UTC m=+1181.848060291" Oct 30 14:26:26.323425 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.323385 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:26.329092 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.329055 4968 patch_prober.go:29] interesting pod/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:26:26.329247 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.329113 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:26:26.370449 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.370417 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:26:26.444738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.443237 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-kube-api-access-smhgk\" (UniqueName: \"kubernetes.io/projected/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-kube-api-access-smhgk\") pod \"9e1775f1-3d4b-491c-b2dd-f207e5bd8025\" (UID: \"9e1775f1-3d4b-491c-b2dd-f207e5bd8025\") " Oct 30 14:26:26.444738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.443366 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-plugins-config-volume\") pod \"9e1775f1-3d4b-491c-b2dd-f207e5bd8025\" (UID: \"9e1775f1-3d4b-491c-b2dd-f207e5bd8025\") " Oct 30 14:26:26.444738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.444441 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-plugins-config-volume" pod "9e1775f1-3d4b-491c-b2dd-f207e5bd8025" (UID: "9e1775f1-3d4b-491c-b2dd-f207e5bd8025"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:26.457743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.446015 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-kube-api-access-smhgk" pod "9e1775f1-3d4b-491c-b2dd-f207e5bd8025" (UID: "9e1775f1-3d4b-491c-b2dd-f207e5bd8025"). InnerVolumeSpecName "kube-api-access-smhgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:26.545550 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.545514 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smhgk\" (UniqueName: \"kubernetes.io/projected/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-kube-api-access-smhgk\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:26.545550 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:26.545552 4968 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1775f1-3d4b-491c-b2dd-f207e5bd8025-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.162448 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.162406 4968 generic.go:356] "Generic (PLEG): container finished" podID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerID="c706d0ae0e30a21cd85d4030607cb29d1872df9d7aa7ca9a834d91f36709d023" exitCode=0 Oct 30 14:26:27.162850 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.162668 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" event={"ID":"550b3932-faa4-4c46-9fa8-80c553a59dba","Type":"ContainerDied","Data":"c706d0ae0e30a21cd85d4030607cb29d1872df9d7aa7ca9a834d91f36709d023"} Oct 30 14:26:27.162850 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.162778 4968 scope.go:122] "RemoveContainer" containerID="867c0583c1961e04c65a0def98aa306bd82727eb2511f8204cdb8ffec7213c68" Oct 30 14:26:27.165164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.165133 4968 generic.go:356] "Generic (PLEG): container finished" podID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerID="6411995df9ab33485e1e1dc0bfcee82487ff5a23a26c11ac02ca44bbfd95af72" exitCode=0 Oct 30 14:26:27.165331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.165302 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" event={"ID":"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90","Type":"ContainerDied","Data":"6411995df9ab33485e1e1dc0bfcee82487ff5a23a26c11ac02ca44bbfd95af72"} Oct 30 14:26:27.167424 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.167398 4968 generic.go:356] "Generic (PLEG): container finished" podID="9e1775f1-3d4b-491c-b2dd-f207e5bd8025" containerID="2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4" exitCode=0 Oct 30 14:26:27.167547 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.167475 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" event={"ID":"9e1775f1-3d4b-491c-b2dd-f207e5bd8025","Type":"ContainerDied","Data":"2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4"} Oct 30 14:26:27.167547 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.167527 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" event={"ID":"9e1775f1-3d4b-491c-b2dd-f207e5bd8025","Type":"ContainerDied","Data":"825abd52851be201e229e6bc3319c88b5c5ab7315fe15d14480839608493df81"} Oct 30 14:26:27.167761 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.167699 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9" Oct 30 14:26:27.197274 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.197252 4968 scope.go:122] "RemoveContainer" containerID="2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4" Oct 30 14:26:27.205902 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.205876 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:26:27.214416 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.213495 4968 scope.go:122] "RemoveContainer" containerID="2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4" Oct 30 14:26:27.214416 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:27.213945 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4\": container with ID starting with 2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4 not found: ID does not exist" containerID="2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4" Oct 30 14:26:27.214416 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.213982 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4"} err="failed to get container status \"2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4\": rpc error: code = NotFound desc = could not find container \"2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4\": container with ID starting with 2ffbf6119d60fd7db5f3fe6ebbbda6b3740c4138494c0afb4c04bb1f7b4013b4 not found: ID does not exist" Oct 30 14:26:27.233600 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.233571 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9"] Oct 30 14:26:27.246896 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.246867 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bf8266-6f748d977c-hmqd9"] Oct 30 14:26:27.259794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.259770 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550b3932-faa4-4c46-9fa8-80c553a59dba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550b3932-faa4-4c46-9fa8-80c553a59dba-istiod-ca-cert\") pod \"550b3932-faa4-4c46-9fa8-80c553a59dba\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " Oct 30 14:26:27.259917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.259822 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-credential-socket\") pod \"550b3932-faa4-4c46-9fa8-80c553a59dba\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " Oct 30 14:26:27.259917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.259904 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-envoy\") pod \"550b3932-faa4-4c46-9fa8-80c553a59dba\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " Oct 30 14:26:27.260031 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.259973 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-data\") pod \"550b3932-faa4-4c46-9fa8-80c553a59dba\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " Oct 30 14:26:27.260031 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.260022 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-certs\") pod \"550b3932-faa4-4c46-9fa8-80c553a59dba\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " Oct 30 14:26:27.260130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.260069 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-podinfo\") pod \"550b3932-faa4-4c46-9fa8-80c553a59dba\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " Oct 30 14:26:27.260130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.260112 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-token\" (UniqueName: \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-token\") pod \"550b3932-faa4-4c46-9fa8-80c553a59dba\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " Oct 30 14:26:27.260236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.260173 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-socket\") pod \"550b3932-faa4-4c46-9fa8-80c553a59dba\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " Oct 30 14:26:27.260236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.260223 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-kube-api-access-c8cj7\" (UniqueName: \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-kube-api-access-c8cj7\") pod \"550b3932-faa4-4c46-9fa8-80c553a59dba\" (UID: \"550b3932-faa4-4c46-9fa8-80c553a59dba\") " Oct 30 14:26:27.260236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.260194 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-credential-socket" pod "550b3932-faa4-4c46-9fa8-80c553a59dba" (UID: "550b3932-faa4-4c46-9fa8-80c553a59dba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.260385 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.260289 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-data" pod "550b3932-faa4-4c46-9fa8-80c553a59dba" (UID: "550b3932-faa4-4c46-9fa8-80c553a59dba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.260385 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.260339 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-certs" pod "550b3932-faa4-4c46-9fa8-80c553a59dba" (UID: "550b3932-faa4-4c46-9fa8-80c553a59dba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.260493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.260428 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-socket" pod "550b3932-faa4-4c46-9fa8-80c553a59dba" (UID: "550b3932-faa4-4c46-9fa8-80c553a59dba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.260560 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.260117 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550b3932-faa4-4c46-9fa8-80c553a59dba-istiod-ca-cert" pod "550b3932-faa4-4c46-9fa8-80c553a59dba" (UID: "550b3932-faa4-4c46-9fa8-80c553a59dba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:27.261177 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.261154 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.261259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.261183 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.261259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.261200 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550b3932-faa4-4c46-9fa8-80c553a59dba-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.261259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.261216 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.261259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.261230 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.262500 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.262471 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-podinfo" pod "550b3932-faa4-4c46-9fa8-80c553a59dba" (UID: "550b3932-faa4-4c46-9fa8-80c553a59dba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:27.262599 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.262586 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-envoy" pod "550b3932-faa4-4c46-9fa8-80c553a59dba" (UID: "550b3932-faa4-4c46-9fa8-80c553a59dba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.262767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.262698 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-token" pod "550b3932-faa4-4c46-9fa8-80c553a59dba" (UID: "550b3932-faa4-4c46-9fa8-80c553a59dba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:27.262767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.262758 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-kube-api-access-c8cj7" pod "550b3932-faa4-4c46-9fa8-80c553a59dba" (UID: "550b3932-faa4-4c46-9fa8-80c553a59dba"). InnerVolumeSpecName "kube-api-access-c8cj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:27.351047 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.351007 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:26:27.362062 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.362037 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8cj7\" (UniqueName: \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-kube-api-access-c8cj7\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.362062 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.362061 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.362212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.362072 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.362212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.362080 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550b3932-faa4-4c46-9fa8-80c553a59dba-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.463096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463018 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-kube-api-access-5bczx\" (UniqueName: \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-kube-api-access-5bczx\") pod \"52388676-c0d2-4229-86ac-ad7763946b31\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " Oct 30 14:26:27.463096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463067 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-data\") pod \"52388676-c0d2-4229-86ac-ad7763946b31\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " Oct 30 14:26:27.463096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463091 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52388676-c0d2-4229-86ac-ad7763946b31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52388676-c0d2-4229-86ac-ad7763946b31-istiod-ca-cert\") pod \"52388676-c0d2-4229-86ac-ad7763946b31\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " Oct 30 14:26:27.463376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463133 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52388676-c0d2-4229-86ac-ad7763946b31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52388676-c0d2-4229-86ac-ad7763946b31-istio-podinfo\") pod \"52388676-c0d2-4229-86ac-ad7763946b31\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " Oct 30 14:26:27.463376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463178 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-envoy\") pod \"52388676-c0d2-4229-86ac-ad7763946b31\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " Oct 30 14:26:27.463376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463235 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-socket\") pod \"52388676-c0d2-4229-86ac-ad7763946b31\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " Oct 30 14:26:27.463376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463326 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-istio-token\" (UniqueName: \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-istio-token\") pod \"52388676-c0d2-4229-86ac-ad7763946b31\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " Oct 30 14:26:27.463576 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463377 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-data" pod "52388676-c0d2-4229-86ac-ad7763946b31" (UID: "52388676-c0d2-4229-86ac-ad7763946b31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.463576 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463386 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-certs\") pod \"52388676-c0d2-4229-86ac-ad7763946b31\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " Oct 30 14:26:27.463576 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463469 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-credential-socket\") pod \"52388676-c0d2-4229-86ac-ad7763946b31\" (UID: \"52388676-c0d2-4229-86ac-ad7763946b31\") " Oct 30 14:26:27.463729 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463645 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-socket" pod "52388676-c0d2-4229-86ac-ad7763946b31" (UID: "52388676-c0d2-4229-86ac-ad7763946b31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.463729 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463657 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52388676-c0d2-4229-86ac-ad7763946b31-istiod-ca-cert" pod "52388676-c0d2-4229-86ac-ad7763946b31" (UID: "52388676-c0d2-4229-86ac-ad7763946b31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:27.463873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463824 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-credential-socket" pod "52388676-c0d2-4229-86ac-ad7763946b31" (UID: "52388676-c0d2-4229-86ac-ad7763946b31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.463873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.463852 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-certs" pod "52388676-c0d2-4229-86ac-ad7763946b31" (UID: "52388676-c0d2-4229-86ac-ad7763946b31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.464087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.464068 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.464164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.464089 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52388676-c0d2-4229-86ac-ad7763946b31-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.464164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.464099 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.464164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.464108 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.464164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.464117 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.465369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.465346 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-kube-api-access-5bczx" pod "52388676-c0d2-4229-86ac-ad7763946b31" (UID: "52388676-c0d2-4229-86ac-ad7763946b31"). InnerVolumeSpecName "kube-api-access-5bczx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:27.465770 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.465746 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-istio-token" pod "52388676-c0d2-4229-86ac-ad7763946b31" (UID: "52388676-c0d2-4229-86ac-ad7763946b31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:27.465858 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.465816 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52388676-c0d2-4229-86ac-ad7763946b31-istio-podinfo" pod "52388676-c0d2-4229-86ac-ad7763946b31" (UID: "52388676-c0d2-4229-86ac-ad7763946b31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:27.465858 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.465821 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-envoy" pod "52388676-c0d2-4229-86ac-ad7763946b31" (UID: "52388676-c0d2-4229-86ac-ad7763946b31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.565876 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.565057 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.565876 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.565092 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bczx\" (UniqueName: \"kubernetes.io/projected/52388676-c0d2-4229-86ac-ad7763946b31-kube-api-access-5bczx\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.565876 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.565109 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52388676-c0d2-4229-86ac-ad7763946b31-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.565876 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.565123 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52388676-c0d2-4229-86ac-ad7763946b31-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.716053 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.715992 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:26:27.771120 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.771093 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-kube-api-access-7hpv4\" (UniqueName: \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-kube-api-access-7hpv4\") pod \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " Oct 30 14:26:27.771295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.771130 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-envoy\") pod \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " Oct 30 14:26:27.771295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.771181 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-token\" (UniqueName: \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-token\") pod \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " Oct 30 14:26:27.771295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.771283 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istiod-ca-cert\") pod \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " Oct 30 14:26:27.771598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.771311 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-credential-socket\") pod \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " Oct 30 14:26:27.771598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.771356 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-data\") pod \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " Oct 30 14:26:27.771598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.771422 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-certs\") pod \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " Oct 30 14:26:27.771598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.771462 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-podinfo\") pod \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " Oct 30 14:26:27.771598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.771496 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-socket\") pod \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\" (UID: \"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90\") " Oct 30 14:26:27.771924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.771664 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-credential-socket" pod "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" (UID: "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.771924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.771872 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-socket" pod "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" (UID: "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.772049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.772025 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-certs" pod "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" (UID: "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.772114 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.772049 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-data" pod "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" (UID: "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.772170 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.772137 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.772170 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.772155 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.772467 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.772412 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istiod-ca-cert" pod "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" (UID: "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:27.774337 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.774315 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-kube-api-access-7hpv4" pod "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" (UID: "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90"). InnerVolumeSpecName "kube-api-access-7hpv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:27.774426 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.774317 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-podinfo" pod "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" (UID: "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:27.775342 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.775317 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-token" pod "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" (UID: "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:27.775436 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.775381 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-envoy" pod "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" (UID: "010bc0ef-5c41-40b4-b6a7-ee2b49eddd90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:27.873289 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.873251 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.873289 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.873289 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.873532 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.873314 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.873532 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.873329 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hpv4\" (UniqueName: \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-kube-api-access-7hpv4\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.873532 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.873344 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.873532 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.873357 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.873532 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.873368 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:27.922824 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.922789 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j"] Oct 30 14:26:27.923032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.923006 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" podUID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" containerName="istio-proxy" containerID="cri-o://a2f946bdc08cd5e8300373cee2f65de4f0692c22abf3fb3fb134edf9a522b513" gracePeriod=30 Oct 30 14:26:27.931577 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.931551 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl"] Oct 30 14:26:27.931790 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.931766 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerName="istio-proxy" containerID="cri-o://287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0" gracePeriod=30 Oct 30 14:26:27.943383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.943357 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb"] Oct 30 14:26:27.943585 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:27.943563 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" containerID="cri-o://f5ec0fc9a536c00189405058d8cff95084f3d57d6e9b55c6d876ab4345b3808a" gracePeriod=30 Oct 30 14:26:28.125166 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.125129 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:26:28.176988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.176954 4968 generic.go:356] "Generic (PLEG): container finished" podID="52388676-c0d2-4229-86ac-ad7763946b31" containerID="338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47" exitCode=0 Oct 30 14:26:28.177509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.177016 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" Oct 30 14:26:28.177509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.177045 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" event={"ID":"52388676-c0d2-4229-86ac-ad7763946b31","Type":"ContainerDied","Data":"338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47"} Oct 30 14:26:28.177509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.177116 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc" event={"ID":"52388676-c0d2-4229-86ac-ad7763946b31","Type":"ContainerDied","Data":"1a6e072e73642a4d70822fe493dc550d35f25978ed885852d70dcad9be9bc8b7"} Oct 30 14:26:28.177509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.177150 4968 scope.go:122] "RemoveContainer" containerID="338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47" Oct 30 14:26:28.178860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.178834 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" event={"ID":"550b3932-faa4-4c46-9fa8-80c553a59dba","Type":"ContainerDied","Data":"1306177b77dd77d9f133b6707ec1e90e5c68cb73b80e4b4ed3a6f9b17867cd2e"} Oct 30 14:26:28.178965 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.178870 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh" Oct 30 14:26:28.181283 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.181255 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" event={"ID":"010bc0ef-5c41-40b4-b6a7-ee2b49eddd90","Type":"ContainerDied","Data":"9d2b162291ad325d345b813b8e12eeff39fac622f450ef3ddd436e7a74db00a0"} Oct 30 14:26:28.181400 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.181374 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb" Oct 30 14:26:28.186182 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.186156 4968 generic.go:356] "Generic (PLEG): container finished" podID="2118e800-48fe-4900-a656-247cc5fa085a" containerID="a4454be6f86b42b9449b6003a2a589fc6efc1c83ad13cce70b526079c2158bbd" exitCode=0 Oct 30 14:26:28.186587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.186476 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" event={"ID":"2118e800-48fe-4900-a656-247cc5fa085a","Type":"ContainerDied","Data":"a4454be6f86b42b9449b6003a2a589fc6efc1c83ad13cce70b526079c2158bbd"} Oct 30 14:26:28.197334 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.197315 4968 scope.go:122] "RemoveContainer" containerID="338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47" Oct 30 14:26:28.197619 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:28.197598 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47\": container with ID starting with 338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47 not found: ID does not exist" containerID="338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47" Oct 30 14:26:28.197619 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.197630 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47"} err="failed to get container status \"338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47\": rpc error: code = NotFound desc = could not find container \"338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47\": container with ID starting with 338166a5bed213275ba8c97b365512e5b040d8769a0f64c62f95b545cde72b47 not found: ID does not exist" Oct 30 14:26:28.197839 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.197653 4968 scope.go:122] "RemoveContainer" containerID="c706d0ae0e30a21cd85d4030607cb29d1872df9d7aa7ca9a834d91f36709d023" Oct 30 14:26:28.209816 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.209784 4968 scope.go:122] "RemoveContainer" containerID="6411995df9ab33485e1e1dc0bfcee82487ff5a23a26c11ac02ca44bbfd95af72" Oct 30 14:26:28.251201 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.251174 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc"] Oct 30 14:26:28.255342 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.255313 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-hbjbc"] Oct 30 14:26:28.317368 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.317338 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb"] Oct 30 14:26:28.362477 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.362447 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-jbtzb"] Oct 30 14:26:28.369203 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.369163 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh"] Oct 30 14:26:28.379507 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.379482 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-24dfd7-openshift-default-67746fb97c-zk9fh"] Oct 30 14:26:28.588417 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.586547 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" path="/var/lib/kubelet/pods/010bc0ef-5c41-40b4-b6a7-ee2b49eddd90/volumes" Oct 30 14:26:28.588417 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.587255 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52388676-c0d2-4229-86ac-ad7763946b31" path="/var/lib/kubelet/pods/52388676-c0d2-4229-86ac-ad7763946b31/volumes" Oct 30 14:26:28.588417 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.587850 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" path="/var/lib/kubelet/pods/550b3932-faa4-4c46-9fa8-80c553a59dba/volumes" Oct 30 14:26:28.589565 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.589109 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1775f1-3d4b-491c-b2dd-f207e5bd8025" path="/var/lib/kubelet/pods/9e1775f1-3d4b-491c-b2dd-f207e5bd8025/volumes" Oct 30 14:26:28.622438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.622412 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:26:28.708964 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.708938 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:26:28.709382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.709359 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-data\") pod \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " Oct 30 14:26:28.709500 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.709396 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-socket\") pod \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " Oct 30 14:26:28.709500 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.709460 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-kube-api-access-5cjnc\" (UniqueName: \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-kube-api-access-5cjnc\") pod \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " Oct 30 14:26:28.709500 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.709495 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-podinfo\") pod \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " Oct 30 14:26:28.709660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.709549 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-certs\") pod \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " Oct 30 14:26:28.709660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.709609 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-token\" (UniqueName: \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-token\") pod \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " Oct 30 14:26:28.709774 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.709656 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-data" pod "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" (UID: "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:28.709774 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.709693 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-envoy\") pod \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " Oct 30 14:26:28.709774 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.709742 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-credential-socket\") pod \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " Oct 30 14:26:28.709919 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.709786 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istiod-ca-cert\") pod \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\" (UID: \"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e\") " Oct 30 14:26:28.710630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.710094 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-socket" pod "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" (UID: "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:28.710630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.710347 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.710630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.710369 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.710630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.710385 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istiod-ca-cert" pod "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" (UID: "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:28.711298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.711186 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-credential-socket" pod "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" (UID: "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:28.711298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.711264 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-certs" pod "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" (UID: "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:28.718577 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.718553 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-kube-api-access-5cjnc" pod "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" (UID: "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e"). InnerVolumeSpecName "kube-api-access-5cjnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:28.718901 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.718860 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-envoy" pod "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" (UID: "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:28.719006 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.718912 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-token" pod "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" (UID: "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:28.720336 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.720312 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-podinfo" pod "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" (UID: "a3caeb10-f0dd-4a43-988c-0f3b8ff8754e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:28.822743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.816172 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-kube-api-access-ss4vk\" (UniqueName: \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-kube-api-access-ss4vk\") pod \"2118e800-48fe-4900-a656-247cc5fa085a\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " Oct 30 14:26:28.826740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.826396 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-envoy\") pod \"2118e800-48fe-4900-a656-247cc5fa085a\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.826999 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2118e800-48fe-4900-a656-247cc5fa085a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2118e800-48fe-4900-a656-247cc5fa085a-istio-podinfo\") pod \"2118e800-48fe-4900-a656-247cc5fa085a\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.827574 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2118e800-48fe-4900-a656-247cc5fa085a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2118e800-48fe-4900-a656-247cc5fa085a-istiod-ca-cert\") pod \"2118e800-48fe-4900-a656-247cc5fa085a\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.827965 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2118e800-48fe-4900-a656-247cc5fa085a-istiod-ca-cert" pod "2118e800-48fe-4900-a656-247cc5fa085a" (UID: "2118e800-48fe-4900-a656-247cc5fa085a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828034 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-data\") pod \"2118e800-48fe-4900-a656-247cc5fa085a\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828109 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-credential-socket\") pod \"2118e800-48fe-4900-a656-247cc5fa085a\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828237 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-socket\") pod \"2118e800-48fe-4900-a656-247cc5fa085a\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828287 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-certs\") pod \"2118e800-48fe-4900-a656-247cc5fa085a\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828322 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-istio-token\" (UniqueName: \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-istio-token\") pod \"2118e800-48fe-4900-a656-247cc5fa085a\" (UID: \"2118e800-48fe-4900-a656-247cc5fa085a\") " Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828868 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828889 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828904 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2118e800-48fe-4900-a656-247cc5fa085a-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828918 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828931 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828944 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828957 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cjnc\" (UniqueName: \"kubernetes.io/projected/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-kube-api-access-5cjnc\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.828984 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.829468 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-credential-socket" pod "2118e800-48fe-4900-a656-247cc5fa085a" (UID: "2118e800-48fe-4900-a656-247cc5fa085a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:28.829794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.829677 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-data" pod "2118e800-48fe-4900-a656-247cc5fa085a" (UID: "2118e800-48fe-4900-a656-247cc5fa085a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:28.832085 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.831845 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-socket" pod "2118e800-48fe-4900-a656-247cc5fa085a" (UID: "2118e800-48fe-4900-a656-247cc5fa085a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:28.832085 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.832056 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-certs" pod "2118e800-48fe-4900-a656-247cc5fa085a" (UID: "2118e800-48fe-4900-a656-247cc5fa085a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:28.842170 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.842128 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2118e800-48fe-4900-a656-247cc5fa085a-istio-podinfo" pod "2118e800-48fe-4900-a656-247cc5fa085a" (UID: "2118e800-48fe-4900-a656-247cc5fa085a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:28.844196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.844158 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-istio-token" pod "2118e800-48fe-4900-a656-247cc5fa085a" (UID: "2118e800-48fe-4900-a656-247cc5fa085a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:28.845164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.845130 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-kube-api-access-ss4vk" pod "2118e800-48fe-4900-a656-247cc5fa085a" (UID: "2118e800-48fe-4900-a656-247cc5fa085a"). InnerVolumeSpecName "kube-api-access-ss4vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:28.847092 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.847052 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-envoy" pod "2118e800-48fe-4900-a656-247cc5fa085a" (UID: "2118e800-48fe-4900-a656-247cc5fa085a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:28.929880 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.929840 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.929880 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.929880 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.929880 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.929895 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.929880 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.929909 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss4vk\" (UniqueName: \"kubernetes.io/projected/2118e800-48fe-4900-a656-247cc5fa085a-kube-api-access-ss4vk\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.930578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.929924 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.930578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.929937 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2118e800-48fe-4900-a656-247cc5fa085a-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.930578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.929950 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:28.930578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:28.929966 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2118e800-48fe-4900-a656-247cc5fa085a-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:29.203681 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.203642 4968 generic.go:356] "Generic (PLEG): container finished" podID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerID="a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97" exitCode=0 Oct 30 14:26:29.204355 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.204329 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" event={"ID":"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e","Type":"ContainerDied","Data":"a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97"} Oct 30 14:26:29.204512 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.204495 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" event={"ID":"a3caeb10-f0dd-4a43-988c-0f3b8ff8754e","Type":"ContainerDied","Data":"984c52b083a7b1e83cf32c53f8487512ba4d637c01b00e48ff0777fe43244a87"} Oct 30 14:26:29.204631 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.204617 4968 scope.go:122] "RemoveContainer" containerID="a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97" Oct 30 14:26:29.204893 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.204874 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf" Oct 30 14:26:29.219544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.217331 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" event={"ID":"2118e800-48fe-4900-a656-247cc5fa085a","Type":"ContainerDied","Data":"2e01c5c40a0ade74d1b986f2ce922296e210bc9b770d857248cbc7f5c2a8e613"} Oct 30 14:26:29.219544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.217430 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5" Oct 30 14:26:29.243251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.243167 4968 scope.go:122] "RemoveContainer" containerID="dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9" Oct 30 14:26:29.262883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.262857 4968 scope.go:122] "RemoveContainer" containerID="a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97" Oct 30 14:26:29.263214 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:29.263190 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97\": container with ID starting with a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97 not found: ID does not exist" containerID="a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97" Oct 30 14:26:29.263302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.263223 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97"} err="failed to get container status \"a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97\": rpc error: code = NotFound desc = could not find container \"a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97\": container with ID starting with a270cf291fa250438b1a84cda78502ab0a7b99e5860d550b2e2cf3f609cc2d97 not found: ID does not exist" Oct 30 14:26:29.263302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.263246 4968 scope.go:122] "RemoveContainer" containerID="dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9" Oct 30 14:26:29.263548 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:29.263534 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9\": container with ID starting with dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9 not found: ID does not exist" containerID="dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9" Oct 30 14:26:29.263596 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.263552 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9"} err="failed to get container status \"dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9\": rpc error: code = NotFound desc = could not find container \"dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9\": container with ID starting with dfd02d607fe6a73d43e52924c3a04b432b152391c42d0331d508247cecd5a0c9 not found: ID does not exist" Oct 30 14:26:29.263596 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.263565 4968 scope.go:122] "RemoveContainer" containerID="a4454be6f86b42b9449b6003a2a589fc6efc1c83ad13cce70b526079c2158bbd" Oct 30 14:26:29.300765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.299922 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5"] Oct 30 14:26:29.308817 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.308780 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-hndq5"] Oct 30 14:26:29.355318 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.355262 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf"] Oct 30 14:26:29.360247 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:29.360219 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-nnkhf"] Oct 30 14:26:30.519725 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:30.519689 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2118e800-48fe-4900-a656-247cc5fa085a" path="/var/lib/kubelet/pods/2118e800-48fe-4900-a656-247cc5fa085a/volumes" Oct 30 14:26:30.520198 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:30.520182 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" path="/var/lib/kubelet/pods/a3caeb10-f0dd-4a43-988c-0f3b8ff8754e/volumes" Oct 30 14:26:32.246127 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.246094 4968 generic.go:356] "Generic (PLEG): container finished" podID="285f4c10-d262-43f5-a3b8-370e4bfe4d04" containerID="37d00805a50e7fa7a6f1fddf7621dfc0123671ef817feeee64b96b0c73aed900" exitCode=0 Oct 30 14:26:32.246449 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.246164 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" event={"ID":"285f4c10-d262-43f5-a3b8-370e4bfe4d04","Type":"ContainerDied","Data":"37d00805a50e7fa7a6f1fddf7621dfc0123671ef817feeee64b96b0c73aed900"} Oct 30 14:26:32.246449 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.246217 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" event={"ID":"285f4c10-d262-43f5-a3b8-370e4bfe4d04","Type":"ContainerDied","Data":"95ad0eef4f7772a9f401f1467aa80475e10fe55bc6154bd795caf118b8d7fc4c"} Oct 30 14:26:32.246449 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.246234 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95ad0eef4f7772a9f401f1467aa80475e10fe55bc6154bd795caf118b8d7fc4c" Oct 30 14:26:32.258097 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.258076 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:32.386369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386335 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-token\" (UniqueName: \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-token\") pod \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " Oct 30 14:26:32.386563 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386387 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istiod-ca-cert\") pod \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " Oct 30 14:26:32.386563 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386429 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-credential-socket\") pod \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " Oct 30 14:26:32.386563 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386487 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-socket\") pod \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " Oct 30 14:26:32.386563 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386510 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-data\") pod \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " Oct 30 14:26:32.386813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386621 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-kube-api-access-rl8vb\" (UniqueName: \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-kube-api-access-rl8vb\") pod \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " Oct 30 14:26:32.386813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386698 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-podinfo\") pod \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " Oct 30 14:26:32.386813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386760 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-envoy\") pod \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " Oct 30 14:26:32.386813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386809 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-certs\") pod \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\" (UID: \"285f4c10-d262-43f5-a3b8-370e4bfe4d04\") " Oct 30 14:26:32.387009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386901 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-socket" pod "285f4c10-d262-43f5-a3b8-370e4bfe4d04" (UID: "285f4c10-d262-43f5-a3b8-370e4bfe4d04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:32.387009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386911 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-data" pod "285f4c10-d262-43f5-a3b8-370e4bfe4d04" (UID: "285f4c10-d262-43f5-a3b8-370e4bfe4d04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:32.387009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386932 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-credential-socket" pod "285f4c10-d262-43f5-a3b8-370e4bfe4d04" (UID: "285f4c10-d262-43f5-a3b8-370e4bfe4d04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:32.387009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.386965 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istiod-ca-cert" pod "285f4c10-d262-43f5-a3b8-370e4bfe4d04" (UID: "285f4c10-d262-43f5-a3b8-370e4bfe4d04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:32.387195 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.387177 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-certs" pod "285f4c10-d262-43f5-a3b8-370e4bfe4d04" (UID: "285f4c10-d262-43f5-a3b8-370e4bfe4d04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:32.387394 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.387339 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:32.387394 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.387372 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:32.387394 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.387387 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:32.387394 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.387400 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:32.387688 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.387412 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:32.388851 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.388826 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-token" pod "285f4c10-d262-43f5-a3b8-370e4bfe4d04" (UID: "285f4c10-d262-43f5-a3b8-370e4bfe4d04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:32.389194 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.389164 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-podinfo" pod "285f4c10-d262-43f5-a3b8-370e4bfe4d04" (UID: "285f4c10-d262-43f5-a3b8-370e4bfe4d04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:32.389300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.389276 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-kube-api-access-rl8vb" pod "285f4c10-d262-43f5-a3b8-370e4bfe4d04" (UID: "285f4c10-d262-43f5-a3b8-370e4bfe4d04"). InnerVolumeSpecName "kube-api-access-rl8vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:32.390032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.390011 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-envoy" pod "285f4c10-d262-43f5-a3b8-370e4bfe4d04" (UID: "285f4c10-d262-43f5-a3b8-370e4bfe4d04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:32.488659 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.488619 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl8vb\" (UniqueName: \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-kube-api-access-rl8vb\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:32.488659 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.488646 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:32.488659 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.488656 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:32.488659 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:32.488665 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285f4c10-d262-43f5-a3b8-370e4bfe4d04-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.266013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.265894 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:26:33.267760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.267515 4968 generic.go:356] "Generic (PLEG): container finished" podID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerID="287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0" exitCode=0 Oct 30 14:26:33.267760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.267611 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" event={"ID":"57f7ff69-5fde-4505-ae97-bb9ed86bda5d","Type":"ContainerDied","Data":"287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0"} Oct 30 14:26:33.267760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.267682 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" event={"ID":"57f7ff69-5fde-4505-ae97-bb9ed86bda5d","Type":"ContainerDied","Data":"18076ccf34b126c689adba97104f88b95ec77ec5e604e0cd701293143181a176"} Oct 30 14:26:33.267760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.267709 4968 scope.go:122] "RemoveContainer" containerID="287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0" Oct 30 14:26:33.273839 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.272054 4968 generic.go:356] "Generic (PLEG): container finished" podID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerID="f5ec0fc9a536c00189405058d8cff95084f3d57d6e9b55c6d876ab4345b3808a" exitCode=0 Oct 30 14:26:33.273839 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.272211 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" event={"ID":"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad","Type":"ContainerDied","Data":"f5ec0fc9a536c00189405058d8cff95084f3d57d6e9b55c6d876ab4345b3808a"} Oct 30 14:26:33.274003 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.273977 4968 generic.go:356] "Generic (PLEG): container finished" podID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" containerID="a2f946bdc08cd5e8300373cee2f65de4f0692c22abf3fb3fb134edf9a522b513" exitCode=0 Oct 30 14:26:33.274237 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.274214 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt" Oct 30 14:26:33.275172 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.275045 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" event={"ID":"4c04a6cc-c66a-4671-beea-8df9ac3114cb","Type":"ContainerDied","Data":"a2f946bdc08cd5e8300373cee2f65de4f0692c22abf3fb3fb134edf9a522b513"} Oct 30 14:26:33.283067 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.283043 4968 scope.go:122] "RemoveContainer" containerID="287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0" Oct 30 14:26:33.283520 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:33.283472 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0\": container with ID starting with 287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0 not found: ID does not exist" containerID="287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0" Oct 30 14:26:33.283632 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.283513 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0"} err="failed to get container status \"287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0\": rpc error: code = NotFound desc = could not find container \"287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0\": container with ID starting with 287cd177e89211a96e7f51ba9cf34409c3f861acf33c48d9d2e9e53cbc6e3ed0 not found: ID does not exist" Oct 30 14:26:33.335300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.335272 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:26:33.337856 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.337829 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:26:33.361636 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.361586 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt"] Oct 30 14:26:33.366298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.366276 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1d36cf-openshift-default-75ff6dbd7-7t4pt"] Oct 30 14:26:33.400485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400238 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istiod-ca-cert\") pod \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " Oct 30 14:26:33.400485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400313 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-podinfo\") pod \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " Oct 30 14:26:33.400485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400371 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-envoy\") pod \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " Oct 30 14:26:33.400485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400408 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-podinfo\") pod \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " Oct 30 14:26:33.400485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400431 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-kube-api-access-s9zrw\" (UniqueName: \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-kube-api-access-s9zrw\") pod \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " Oct 30 14:26:33.400485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400449 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-kube-api-access-sqv6m\" (UniqueName: \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-kube-api-access-sqv6m\") pod \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " Oct 30 14:26:33.400485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400486 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-credential-socket\") pod \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " Oct 30 14:26:33.401039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400514 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-kube-api-access-twj4b\" (UniqueName: \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-kube-api-access-twj4b\") pod \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " Oct 30 14:26:33.401039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400540 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-socket\") pod \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " Oct 30 14:26:33.401039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400585 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-podinfo\") pod \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " Oct 30 14:26:33.401039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400643 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-socket\") pod \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " Oct 30 14:26:33.401039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400671 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-token\" (UniqueName: \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-token\") pod \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " Oct 30 14:26:33.401039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400708 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-token\" (UniqueName: \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-token\") pod \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " Oct 30 14:26:33.401039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400756 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-certs\") pod \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " Oct 30 14:26:33.401039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400793 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-envoy\") pod \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " Oct 30 14:26:33.401039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400820 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-certs\") pod \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " Oct 30 14:26:33.401039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.400853 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-data\") pod \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " Oct 30 14:26:33.401768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.401463 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-data\") pod \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " Oct 30 14:26:33.401768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.401506 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-certs\") pod \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " Oct 30 14:26:33.401768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.401548 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istiod-ca-cert\") pod \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " Oct 30 14:26:33.401768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.401579 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-data\") pod \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " Oct 30 14:26:33.401768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.401649 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-token\" (UniqueName: \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-token\") pod \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\" (UID: \"4c04a6cc-c66a-4671-beea-8df9ac3114cb\") " Oct 30 14:26:33.401768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.401688 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-envoy\") pod \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " Oct 30 14:26:33.401768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.401752 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istiod-ca-cert\") pod \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " Oct 30 14:26:33.402109 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.401777 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-socket\") pod \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " Oct 30 14:26:33.402109 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.401850 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-credential-socket\") pod \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\" (UID: \"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad\") " Oct 30 14:26:33.402109 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.401978 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-credential-socket\") pod \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\" (UID: \"57f7ff69-5fde-4505-ae97-bb9ed86bda5d\") " Oct 30 14:26:33.402533 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.402508 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-data" pod "57f7ff69-5fde-4505-ae97-bb9ed86bda5d" (UID: "57f7ff69-5fde-4505-ae97-bb9ed86bda5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.402905 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.402868 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-socket" pod "57f7ff69-5fde-4505-ae97-bb9ed86bda5d" (UID: "57f7ff69-5fde-4505-ae97-bb9ed86bda5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.404387 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-data" pod "4c04a6cc-c66a-4671-beea-8df9ac3114cb" (UID: "4c04a6cc-c66a-4671-beea-8df9ac3114cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.404567 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-certs" pod "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" (UID: "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.404964 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istiod-ca-cert" pod "57f7ff69-5fde-4505-ae97-bb9ed86bda5d" (UID: "57f7ff69-5fde-4505-ae97-bb9ed86bda5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.405185 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-data" pod "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" (UID: "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.405685 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-socket" pod "4c04a6cc-c66a-4671-beea-8df9ac3114cb" (UID: "4c04a6cc-c66a-4671-beea-8df9ac3114cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.407357 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istiod-ca-cert" pod "4c04a6cc-c66a-4671-beea-8df9ac3114cb" (UID: "4c04a6cc-c66a-4671-beea-8df9ac3114cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.407799 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-podinfo" pod "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" (UID: "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.408008 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-certs" pod "57f7ff69-5fde-4505-ae97-bb9ed86bda5d" (UID: "57f7ff69-5fde-4505-ae97-bb9ed86bda5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.408193 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-token" pod "4c04a6cc-c66a-4671-beea-8df9ac3114cb" (UID: "4c04a6cc-c66a-4671-beea-8df9ac3114cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.408482 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istiod-ca-cert" pod "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" (UID: "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.408854 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-socket" pod "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" (UID: "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409033 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-credential-socket" pod "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" (UID: "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409242 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-credential-socket" pod "57f7ff69-5fde-4505-ae97-bb9ed86bda5d" (UID: "57f7ff69-5fde-4505-ae97-bb9ed86bda5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409466 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409487 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409502 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409514 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409527 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409540 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409552 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409562 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409573 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409631 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409645 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409658 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409673 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409686 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.410374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.409702 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.411958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.410574 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-certs" pod "4c04a6cc-c66a-4671-beea-8df9ac3114cb" (UID: "4c04a6cc-c66a-4671-beea-8df9ac3114cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.411958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.410775 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-credential-socket" pod "4c04a6cc-c66a-4671-beea-8df9ac3114cb" (UID: "4c04a6cc-c66a-4671-beea-8df9ac3114cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.411958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.411359 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-kube-api-access-twj4b" pod "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" (UID: "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad"). InnerVolumeSpecName "kube-api-access-twj4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:33.412408 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.412367 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-envoy" pod "57f7ff69-5fde-4505-ae97-bb9ed86bda5d" (UID: "57f7ff69-5fde-4505-ae97-bb9ed86bda5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.412705 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.412548 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-envoy" pod "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" (UID: "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.412705 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.412607 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-podinfo" pod "4c04a6cc-c66a-4671-beea-8df9ac3114cb" (UID: "4c04a6cc-c66a-4671-beea-8df9ac3114cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:33.412969 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.412938 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-podinfo" pod "57f7ff69-5fde-4505-ae97-bb9ed86bda5d" (UID: "57f7ff69-5fde-4505-ae97-bb9ed86bda5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:33.413212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.413153 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-token" pod "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" (UID: "432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:33.413618 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.413589 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-kube-api-access-sqv6m" pod "57f7ff69-5fde-4505-ae97-bb9ed86bda5d" (UID: "57f7ff69-5fde-4505-ae97-bb9ed86bda5d"). InnerVolumeSpecName "kube-api-access-sqv6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:33.413980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.413952 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-token" pod "57f7ff69-5fde-4505-ae97-bb9ed86bda5d" (UID: "57f7ff69-5fde-4505-ae97-bb9ed86bda5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:33.414237 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.414201 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-envoy" pod "4c04a6cc-c66a-4671-beea-8df9ac3114cb" (UID: "4c04a6cc-c66a-4671-beea-8df9ac3114cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:33.415279 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.415259 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-kube-api-access-s9zrw" pod "4c04a6cc-c66a-4671-beea-8df9ac3114cb" (UID: "4c04a6cc-c66a-4671-beea-8df9ac3114cb"). InnerVolumeSpecName "kube-api-access-s9zrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:33.510968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510881 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.510968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510907 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twj4b\" (UniqueName: \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-kube-api-access-twj4b\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.510968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510920 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.510968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510930 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.510968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510938 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.510968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510947 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c04a6cc-c66a-4671-beea-8df9ac3114cb-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.510968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510955 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.510968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510962 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c04a6cc-c66a-4671-beea-8df9ac3114cb-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.510968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510970 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.510968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510979 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.511430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510987 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9zrw\" (UniqueName: \"kubernetes.io/projected/4c04a6cc-c66a-4671-beea-8df9ac3114cb-kube-api-access-s9zrw\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:33.511430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:33.510996 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqv6m\" (UniqueName: \"kubernetes.io/projected/57f7ff69-5fde-4505-ae97-bb9ed86bda5d-kube-api-access-sqv6m\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:34.283379 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.283350 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl" Oct 30 14:26:34.285013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.284987 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" Oct 30 14:26:34.285013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.285001 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb" event={"ID":"432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad","Type":"ContainerDied","Data":"cb5c489aaab6553529669c4b5de0fec8215e4276116d6803bce68adbbd6175e5"} Oct 30 14:26:34.285234 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.285067 4968 scope.go:122] "RemoveContainer" containerID="f5ec0fc9a536c00189405058d8cff95084f3d57d6e9b55c6d876ab4345b3808a" Oct 30 14:26:34.286849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.286810 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" Oct 30 14:26:34.286961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.286852 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j" event={"ID":"4c04a6cc-c66a-4671-beea-8df9ac3114cb","Type":"ContainerDied","Data":"74db1cdb8d7fe996486658220465f4d38c03e4339a82c3f0122cd73c874f28a6"} Oct 30 14:26:34.297190 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.297079 4968 scope.go:122] "RemoveContainer" containerID="a2f946bdc08cd5e8300373cee2f65de4f0692c22abf3fb3fb134edf9a522b513" Oct 30 14:26:34.343108 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.343047 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl"] Oct 30 14:26:34.352629 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.352607 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-75gtl"] Oct 30 14:26:34.383731 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.383689 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb"] Oct 30 14:26:34.389528 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.389500 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-9tqgb"] Oct 30 14:26:34.423141 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.423109 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j"] Oct 30 14:26:34.431748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.431700 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bf8266-openshift-default-744cb66d9-psn9j"] Oct 30 14:26:34.519251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.519218 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="285f4c10-d262-43f5-a3b8-370e4bfe4d04" path="/var/lib/kubelet/pods/285f4c10-d262-43f5-a3b8-370e4bfe4d04/volumes" Oct 30 14:26:34.519664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.519651 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" path="/var/lib/kubelet/pods/432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad/volumes" Oct 30 14:26:34.520088 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.520075 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" path="/var/lib/kubelet/pods/4c04a6cc-c66a-4671-beea-8df9ac3114cb/volumes" Oct 30 14:26:34.520793 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:34.520778 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" path="/var/lib/kubelet/pods/57f7ff69-5fde-4505-ae97-bb9ed86bda5d/volumes" Oct 30 14:26:35.323351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:35.323317 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:35.336669 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:35.336640 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:36.316481 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:36.316453 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:26:44.660611 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:44.660583 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:26:44.668216 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:44.668192 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:26:45.688116 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:45.688085 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:26:46.868794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.868757 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc"] Oct 30 14:26:46.869400 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869371 4968 cpu_manager.go:414] "containerMap: removing container" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerName="istio-proxy" Oct 30 14:26:46.869400 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869394 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869430 4968 cpu_manager.go:414] "containerMap: removing container" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869438 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869459 4968 cpu_manager.go:414] "containerMap: removing container" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869467 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869479 4968 cpu_manager.go:414] "containerMap: removing container" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869485 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869494 4968 cpu_manager.go:414] "containerMap: removing container" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869500 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869514 4968 cpu_manager.go:414] "containerMap: removing container" podUID="52388676-c0d2-4229-86ac-ad7763946b31" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869521 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52388676-c0d2-4229-86ac-ad7763946b31" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869535 4968 cpu_manager.go:414] "containerMap: removing container" podUID="9e1775f1-3d4b-491c-b2dd-f207e5bd8025" containerName="payload-processing" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869542 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1775f1-3d4b-491c-b2dd-f207e5bd8025" containerName="payload-processing" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869558 4968 cpu_manager.go:414] "containerMap: removing container" podUID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869564 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869577 4968 cpu_manager.go:414] "containerMap: removing container" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869584 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" Oct 30 14:26:46.869589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869597 4968 cpu_manager.go:414] "containerMap: removing container" podUID="285f4c10-d262-43f5-a3b8-370e4bfe4d04" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869605 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="285f4c10-d262-43f5-a3b8-370e4bfe4d04" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869744 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869758 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c04a6cc-c66a-4671-beea-8df9ac3114cb" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869768 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869777 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="285f4c10-d262-43f5-a3b8-370e4bfe4d04" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869786 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869797 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f7ff69-5fde-4505-ae97-bb9ed86bda5d" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869811 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="010bc0ef-5c41-40b4-b6a7-ee2b49eddd90" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869822 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="432f1cdf-16a2-4e08-aa85-9b8c7ab0c4ad" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869831 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="2118e800-48fe-4900-a656-247cc5fa085a" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869842 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="52388676-c0d2-4229-86ac-ad7763946b31" containerName="istio-proxy" Oct 30 14:26:46.870307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.869851 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1775f1-3d4b-491c-b2dd-f207e5bd8025" containerName="payload-processing" Oct 30 14:26:46.876767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.876748 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:46.907212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.907184 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc"] Oct 30 14:26:46.971099 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.971033 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:46.971099 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.971071 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:46.971388 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.971176 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:46.971388 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.971248 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdksj\" (UniqueName: \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-kube-api-access-bdksj\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:46.971505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.971432 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:46.971505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.971482 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:46.971611 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.971577 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:46.971672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.971607 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:46.971748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.971698 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:46.997258 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.997222 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4"] Oct 30 14:26:46.997744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.997703 4968 cpu_manager.go:414] "containerMap: removing container" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" Oct 30 14:26:46.997800 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.997747 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550b3932-faa4-4c46-9fa8-80c553a59dba" containerName="istio-proxy" Oct 30 14:26:46.997800 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.997789 4968 cpu_manager.go:414] "containerMap: removing container" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" Oct 30 14:26:46.997800 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.997795 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" Oct 30 14:26:46.997900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:46.997876 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3caeb10-f0dd-4a43-988c-0f3b8ff8754e" containerName="istio-proxy" Oct 30 14:26:47.001514 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.001492 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.027475 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.027444 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4"] Oct 30 14:26:47.072492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072455 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.072671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072518 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.072671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072544 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.072671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072587 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.072671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072629 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.072671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072663 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.072980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072693 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.072980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072753 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ffd442-9b53-4b73-9af8-2fa54d148b78-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.072980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072805 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdksj\" (UniqueName: \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-kube-api-access-bdksj\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.072980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072832 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.072980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072871 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.072980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072903 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.072980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.072949 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.073776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.073001 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2zb\" (UniqueName: \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-kube-api-access-7t2zb\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.073776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.073028 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.073776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.073261 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.073776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.073037 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.073776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.073327 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.073776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.073363 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.073776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.073411 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.074374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.074140 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.074374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.074148 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.074374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.074261 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.076562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.076543 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.081957 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.081933 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.095782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.095754 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.096639 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.096616 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdksj\" (UniqueName: \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-kube-api-access-bdksj\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.174494 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.174419 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.174494 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.174476 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.174494 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.174495 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.174782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.174520 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ffd442-9b53-4b73-9af8-2fa54d148b78-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.174782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.174543 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.174782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.174566 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.174782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.174617 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2zb\" (UniqueName: \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-kube-api-access-7t2zb\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.174782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.174661 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.174782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.174702 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.175103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.174884 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.175103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.174930 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.175266 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.175243 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.175345 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.175321 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ffd442-9b53-4b73-9af8-2fa54d148b78-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.175443 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.175421 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.176909 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.176883 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.177204 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.177183 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.194286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.194259 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:47.196959 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.196935 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.197154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.197135 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2zb\" (UniqueName: \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-kube-api-access-7t2zb\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.316516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.316484 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:47.521161 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.521139 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc"] Oct 30 14:26:47.526834 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:26:47.526803 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0614bcf9_f822_4d09_acd4_4ae4b55a2461.slice/crio-9c62f3ca7ab55f27b1aa7696d61c098924c74bf5034751d268fc533c5961346d WatchSource:0}: Error finding container 9c62f3ca7ab55f27b1aa7696d61c098924c74bf5034751d268fc533c5961346d: Status 404 returned error can't find the container with id 9c62f3ca7ab55f27b1aa7696d61c098924c74bf5034751d268fc533c5961346d Oct 30 14:26:47.528649 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.528615 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:47.528757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.528680 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:47.528757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.528706 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:47.628278 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:26:47.628249 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ffd442_9b53_4b73_9af8_2fa54d148b78.slice/crio-36f1101fe129c0faf866d21a4c2e4131e0e3d3b8624830eb64e4a6bca9cda711 WatchSource:0}: Error finding container 36f1101fe129c0faf866d21a4c2e4131e0e3d3b8624830eb64e4a6bca9cda711: Status 404 returned error can't find the container with id 36f1101fe129c0faf866d21a4c2e4131e0e3d3b8624830eb64e4a6bca9cda711 Oct 30 14:26:47.628278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.628259 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4"] Oct 30 14:26:47.631400 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.631323 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:47.631499 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.631454 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:47.631547 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:47.631500 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:48.444316 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:48.444266 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" event={"ID":"0614bcf9-f822-4d09-acd4-4ae4b55a2461","Type":"ContainerStarted","Data":"77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46"} Oct 30 14:26:48.444779 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:48.444373 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" event={"ID":"0614bcf9-f822-4d09-acd4-4ae4b55a2461","Type":"ContainerStarted","Data":"9c62f3ca7ab55f27b1aa7696d61c098924c74bf5034751d268fc533c5961346d"} Oct 30 14:26:48.446049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:48.446022 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" event={"ID":"64ffd442-9b53-4b73-9af8-2fa54d148b78","Type":"ContainerStarted","Data":"0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f"} Oct 30 14:26:48.446184 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:48.446063 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" event={"ID":"64ffd442-9b53-4b73-9af8-2fa54d148b78","Type":"ContainerStarted","Data":"36f1101fe129c0faf866d21a4c2e4131e0e3d3b8624830eb64e4a6bca9cda711"} Oct 30 14:26:48.577794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:48.577739 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" podStartSLOduration=2.5777059639999997 podStartE2EDuration="2.577705964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:48.528460969 +0000 UTC m=+1204.153904343" watchObservedRunningTime="2026-10-30 14:26:48.577705964 +0000 UTC m=+1204.203149337" Oct 30 14:26:48.578296 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:48.578268 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" podStartSLOduration=2.57825995 podStartE2EDuration="2.57825995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:48.573171514 +0000 UTC m=+1204.198614889" watchObservedRunningTime="2026-10-30 14:26:48.57825995 +0000 UTC m=+1204.203703324" Oct 30 14:26:49.196627 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:49.196589 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:49.197025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:49.196996 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 14:26:49.197202 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:49.197176 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 14:26:49.322064 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:49.322030 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:49.322425 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:49.322403 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 14:26:49.322597 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:49.322574 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" podUID="64ffd442-9b53-4b73-9af8-2fa54d148b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 14:26:50.196150 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:50.196109 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 14:26:50.196770 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:50.196734 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 14:26:50.318094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:50.318055 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 14:26:50.318399 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:50.318365 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" podUID="64ffd442-9b53-4b73-9af8-2fa54d148b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 14:26:50.465928 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:26:50.465848 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:26:51.196743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:51.195066 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 14:26:51.196743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:51.195142 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 14:26:51.320739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:51.320065 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 14:26:51.320739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:51.320128 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" podUID="64ffd442-9b53-4b73-9af8-2fa54d148b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 14:26:52.196741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:52.196079 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 14:26:52.196741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:52.196140 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 14:26:52.321738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:52.321039 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 14:26:52.321738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:52.321100 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" podUID="64ffd442-9b53-4b73-9af8-2fa54d148b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 14:26:53.197113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:53.197073 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 14:26:53.197755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:53.197682 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 14:26:53.320742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:53.319129 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 14:26:53.320742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:53.319209 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" podUID="64ffd442-9b53-4b73-9af8-2fa54d148b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 14:26:54.195283 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:54.195238 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 14:26:54.195466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:54.195321 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 14:26:54.317142 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:54.317089 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 14:26:54.317646 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:54.317149 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" podUID="64ffd442-9b53-4b73-9af8-2fa54d148b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 14:26:55.199113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.199077 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:55.199427 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.199404 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:55.200305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.200283 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:26:55.324630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.324593 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:55.326594 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.326571 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:55.333507 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.332088 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:26:55.560587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.560502 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt"] Oct 30 14:26:55.565749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.565702 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:55.580929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.580908 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:26:55.580929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.580921 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:26:55.581109 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.580969 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:26:55.581109 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.581002 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:26:55.586081 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.586049 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt"] Oct 30 14:26:55.657461 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.657424 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cb2b3e-5jzgt\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:55.657623 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.657485 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg7rw\" (UniqueName: \"kubernetes.io/projected/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-kube-api-access-bg7rw\") pod \"maas-api-revoke-sub-d5cb2b3e-5jzgt\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:55.658154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.658124 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cb2b3e-5jzgt\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:55.661218 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.661192 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv"] Oct 30 14:26:55.666531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.666512 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.682801 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.682780 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-137f01\"/\"distinct-2-w4-137f01-kserve-self-signed-certs\"" Oct 30 14:26:55.683970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.683937 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv"] Oct 30 14:26:55.759036 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.759003 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/726af6c4-5d67-47dc-8d3a-abbaec756c58-tls-certs\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.759036 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.759037 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-dshm\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.759250 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.759054 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-tmp-dir\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.759250 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.759128 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cb2b3e-5jzgt\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:55.759250 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.759171 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg7rw\" (UniqueName: \"kubernetes.io/projected/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-kube-api-access-bg7rw\") pod \"maas-api-revoke-sub-d5cb2b3e-5jzgt\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:55.759376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.759320 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cb2b3e-5jzgt\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:55.759376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.759350 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-home\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.759443 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.759409 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-model-cache\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.759481 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.759445 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-468vr\" (UniqueName: \"kubernetes.io/projected/726af6c4-5d67-47dc-8d3a-abbaec756c58-kube-api-access-468vr\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.759813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.759795 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cb2b3e-5jzgt\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:55.761622 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.761601 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cb2b3e-5jzgt\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:55.780797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.780774 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg7rw\" (UniqueName: \"kubernetes.io/projected/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-kube-api-access-bg7rw\") pod \"maas-api-revoke-sub-d5cb2b3e-5jzgt\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:55.861745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.860832 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-model-cache\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.861745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.860910 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-468vr\" (UniqueName: \"kubernetes.io/projected/726af6c4-5d67-47dc-8d3a-abbaec756c58-kube-api-access-468vr\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.861745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.860944 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/726af6c4-5d67-47dc-8d3a-abbaec756c58-tls-certs\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.861745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.860970 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-dshm\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.861745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.860994 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-tmp-dir\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.861745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.861206 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-home\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.862118 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.861786 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-home\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.862118 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.862100 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-model-cache\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.864043 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.864015 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-tmp-dir\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.868760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.868277 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-dshm\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.894494 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.894450 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:55.897263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.897233 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/726af6c4-5d67-47dc-8d3a-abbaec756c58-tls-certs\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.897436 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.897399 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-468vr\" (UniqueName: \"kubernetes.io/projected/726af6c4-5d67-47dc-8d3a-abbaec756c58-kube-api-access-468vr\") pod \"distinct-2-w4-137f01-kserve-65d96fff96-dfcqv\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:55.979065 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:55.979036 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:56.203869 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:56.203835 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt"] Oct 30 14:26:56.204694 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:26:56.204669 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2337deb7_cb13_44d4_b3f4_309eb5c7c99d.slice/crio-eab32b890e0753bdf7d623c08010fcbaa7018bbe593b61c2d10af30e1f84d972 WatchSource:0}: Error finding container eab32b890e0753bdf7d623c08010fcbaa7018bbe593b61c2d10af30e1f84d972: Status 404 returned error can't find the container with id eab32b890e0753bdf7d623c08010fcbaa7018bbe593b61c2d10af30e1f84d972 Oct 30 14:26:56.295240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:56.295214 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv"] Oct 30 14:26:56.296133 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:26:56.296103 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod726af6c4_5d67_47dc_8d3a_abbaec756c58.slice/crio-b7f157a1a58c28914b3045898f157656c2a5a5953e69011905da11c0e16a8a38 WatchSource:0}: Error finding container b7f157a1a58c28914b3045898f157656c2a5a5953e69011905da11c0e16a8a38: Status 404 returned error can't find the container with id b7f157a1a58c28914b3045898f157656c2a5a5953e69011905da11c0e16a8a38 Oct 30 14:26:56.532781 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:56.532750 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" event={"ID":"2337deb7-cb13-44d4-b3f4-309eb5c7c99d","Type":"ContainerStarted","Data":"b74ea38d4e69e9fe820cb07a3a6eb20b1fd7660d97c76b97eb58289dadb558a0"} Oct 30 14:26:56.533146 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:56.532808 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" event={"ID":"2337deb7-cb13-44d4-b3f4-309eb5c7c99d","Type":"ContainerStarted","Data":"eab32b890e0753bdf7d623c08010fcbaa7018bbe593b61c2d10af30e1f84d972"} Oct 30 14:26:56.534493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:56.534467 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" event={"ID":"726af6c4-5d67-47dc-8d3a-abbaec756c58","Type":"ContainerStarted","Data":"43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e"} Oct 30 14:26:56.534596 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:56.534500 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" event={"ID":"726af6c4-5d67-47dc-8d3a-abbaec756c58","Type":"ContainerStarted","Data":"b7f157a1a58c28914b3045898f157656c2a5a5953e69011905da11c0e16a8a38"} Oct 30 14:26:56.572412 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:56.572366 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" podStartSLOduration=1.5723535800000001 podStartE2EDuration="1.57235358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:56.570071363 +0000 UTC m=+1212.195514747" watchObservedRunningTime="2026-10-30 14:26:56.57235358 +0000 UTC m=+1212.197796952" Oct 30 14:26:56.608973 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:56.608935 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" podStartSLOduration=1.6089238479999999 podStartE2EDuration="1.608923848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:56.608232673 +0000 UTC m=+1212.233676058" watchObservedRunningTime="2026-10-30 14:26:56.608923848 +0000 UTC m=+1212.234367221" Oct 30 14:26:56.979642 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:56.979607 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:26:57.553264 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:57.553230 4968 generic.go:356] "Generic (PLEG): container finished" podID="2337deb7-cb13-44d4-b3f4-309eb5c7c99d" containerID="b74ea38d4e69e9fe820cb07a3a6eb20b1fd7660d97c76b97eb58289dadb558a0" exitCode=0 Oct 30 14:26:57.553768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:57.553323 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" event={"ID":"2337deb7-cb13-44d4-b3f4-309eb5c7c99d","Type":"ContainerDied","Data":"b74ea38d4e69e9fe820cb07a3a6eb20b1fd7660d97c76b97eb58289dadb558a0"} Oct 30 14:26:58.722352 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:58.722278 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:58.795497 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:58.795455 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-kube-api-access-bg7rw\" (UniqueName: \"kubernetes.io/projected/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-kube-api-access-bg7rw\") pod \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " Oct 30 14:26:58.795743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:58.795526 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-xks-ca\") pod \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " Oct 30 14:26:58.795743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:58.795600 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-openshift-ca\") pod \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\" (UID: \"2337deb7-cb13-44d4-b3f4-309eb5c7c99d\") " Oct 30 14:26:58.796676 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:58.796619 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-openshift-ca" pod "2337deb7-cb13-44d4-b3f4-309eb5c7c99d" (UID: "2337deb7-cb13-44d4-b3f4-309eb5c7c99d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:58.798556 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:58.798533 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-kube-api-access-bg7rw" pod "2337deb7-cb13-44d4-b3f4-309eb5c7c99d" (UID: "2337deb7-cb13-44d4-b3f4-309eb5c7c99d"). InnerVolumeSpecName "kube-api-access-bg7rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:58.802356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:58.802330 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-xks-ca" pod "2337deb7-cb13-44d4-b3f4-309eb5c7c99d" (UID: "2337deb7-cb13-44d4-b3f4-309eb5c7c99d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:26:58.897286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:58.897243 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg7rw\" (UniqueName: \"kubernetes.io/projected/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-kube-api-access-bg7rw\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:58.897286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:58.897284 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:58.897578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:58.897298 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2337deb7-cb13-44d4-b3f4-309eb5c7c99d-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:26:59.364588 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:59.364553 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:26:59.574028 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:59.573999 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" Oct 30 14:26:59.574208 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:59.574038 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt" event={"ID":"2337deb7-cb13-44d4-b3f4-309eb5c7c99d","Type":"ContainerDied","Data":"eab32b890e0753bdf7d623c08010fcbaa7018bbe593b61c2d10af30e1f84d972"} Oct 30 14:26:59.574208 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:26:59.574093 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eab32b890e0753bdf7d623c08010fcbaa7018bbe593b61c2d10af30e1f84d972" Oct 30 14:27:05.980044 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:05.980011 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:27:05.993094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:05.993065 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:27:06.646681 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.646648 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp"] Oct 30 14:27:06.647145 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.647131 4968 cpu_manager.go:414] "containerMap: removing container" podUID="2337deb7-cb13-44d4-b3f4-309eb5c7c99d" containerName="revoke-keys" Oct 30 14:27:06.647187 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.647147 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2337deb7-cb13-44d4-b3f4-309eb5c7c99d" containerName="revoke-keys" Oct 30 14:27:06.647239 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.647230 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="2337deb7-cb13-44d4-b3f4-309eb5c7c99d" containerName="revoke-keys" Oct 30 14:27:06.650761 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.650743 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:06.672360 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.672329 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp"] Oct 30 14:27:06.675461 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.675438 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:27:06.675641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.675504 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:27:06.675803 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.675557 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:27:06.675912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.675626 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:27:06.686603 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.686582 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:27:06.744236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.744198 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghnm\" (UniqueName: \"kubernetes.io/projected/caf26b3d-b7e9-4003-90c9-a9ec050247ae-kube-api-access-bghnm\") pod \"maas-api-revoke-keys-e2e-worker-w1-717995-25vsp\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:06.744400 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.744258 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-717995-25vsp\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:06.744490 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.744461 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-717995-25vsp\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:06.846000 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.845968 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-717995-25vsp\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:06.846197 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.846076 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghnm\" (UniqueName: \"kubernetes.io/projected/caf26b3d-b7e9-4003-90c9-a9ec050247ae-kube-api-access-bghnm\") pod \"maas-api-revoke-keys-e2e-worker-w1-717995-25vsp\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:06.846197 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.846101 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-717995-25vsp\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:06.846780 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.846755 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-717995-25vsp\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:06.848493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.848465 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-717995-25vsp\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:06.867880 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.867849 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghnm\" (UniqueName: \"kubernetes.io/projected/caf26b3d-b7e9-4003-90c9-a9ec050247ae-kube-api-access-bghnm\") pod \"maas-api-revoke-keys-e2e-worker-w1-717995-25vsp\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:06.965080 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:06.965003 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:07.287821 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:07.280221 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp"] Oct 30 14:27:07.287821 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:07.280483 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf26b3d_b7e9_4003_90c9_a9ec050247ae.slice/crio-eaaefd942a2572fa712041bfb66bb5382c6d0d0292e78e7343c1fa24371ee841 WatchSource:0}: Error finding container eaaefd942a2572fa712041bfb66bb5382c6d0d0292e78e7343c1fa24371ee841: Status 404 returned error can't find the container with id eaaefd942a2572fa712041bfb66bb5382c6d0d0292e78e7343c1fa24371ee841 Oct 30 14:27:07.685641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:07.685602 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" event={"ID":"caf26b3d-b7e9-4003-90c9-a9ec050247ae","Type":"ContainerStarted","Data":"ef6a6d0d3bbb573bdcb9a43a4e7173021c921f68dff39506c92cef3719904df7"} Oct 30 14:27:07.685825 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:07.685671 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" event={"ID":"caf26b3d-b7e9-4003-90c9-a9ec050247ae","Type":"ContainerStarted","Data":"eaaefd942a2572fa712041bfb66bb5382c6d0d0292e78e7343c1fa24371ee841"} Oct 30 14:27:07.736873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:07.736814 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" podStartSLOduration=1.7368012259999999 podStartE2EDuration="1.736801226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:07.73263722 +0000 UTC m=+1223.358080593" watchObservedRunningTime="2026-10-30 14:27:07.736801226 +0000 UTC m=+1223.362244599" Oct 30 14:27:08.697526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:08.697494 4968 generic.go:356] "Generic (PLEG): container finished" podID="caf26b3d-b7e9-4003-90c9-a9ec050247ae" containerID="ef6a6d0d3bbb573bdcb9a43a4e7173021c921f68dff39506c92cef3719904df7" exitCode=0 Oct 30 14:27:08.698039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:08.697548 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" event={"ID":"caf26b3d-b7e9-4003-90c9-a9ec050247ae","Type":"ContainerDied","Data":"ef6a6d0d3bbb573bdcb9a43a4e7173021c921f68dff39506c92cef3719904df7"} Oct 30 14:27:09.855372 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:09.855343 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:09.995897 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:09.995813 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf26b3d-b7e9-4003-90c9-a9ec050247ae-kube-api-access-bghnm\" (UniqueName: \"kubernetes.io/projected/caf26b3d-b7e9-4003-90c9-a9ec050247ae-kube-api-access-bghnm\") pod \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " Oct 30 14:27:09.995897 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:09.995873 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-xks-ca\") pod \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " Oct 30 14:27:09.996074 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:09.996035 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-openshift-ca\") pod \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\" (UID: \"caf26b3d-b7e9-4003-90c9-a9ec050247ae\") " Oct 30 14:27:09.996459 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:09.996422 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-openshift-ca" pod "caf26b3d-b7e9-4003-90c9-a9ec050247ae" (UID: "caf26b3d-b7e9-4003-90c9-a9ec050247ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:09.997974 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:09.997951 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-xks-ca" pod "caf26b3d-b7e9-4003-90c9-a9ec050247ae" (UID: "caf26b3d-b7e9-4003-90c9-a9ec050247ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:09.998114 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:09.997992 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf26b3d-b7e9-4003-90c9-a9ec050247ae-kube-api-access-bghnm" pod "caf26b3d-b7e9-4003-90c9-a9ec050247ae" (UID: "caf26b3d-b7e9-4003-90c9-a9ec050247ae"). InnerVolumeSpecName "kube-api-access-bghnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:10.097086 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:10.097049 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bghnm\" (UniqueName: \"kubernetes.io/projected/caf26b3d-b7e9-4003-90c9-a9ec050247ae-kube-api-access-bghnm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:10.097086 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:10.097086 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:10.097300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:10.097099 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf26b3d-b7e9-4003-90c9-a9ec050247ae-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:10.716390 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:10.716351 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" Oct 30 14:27:10.716586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:10.716391 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp" event={"ID":"caf26b3d-b7e9-4003-90c9-a9ec050247ae","Type":"ContainerDied","Data":"eaaefd942a2572fa712041bfb66bb5382c6d0d0292e78e7343c1fa24371ee841"} Oct 30 14:27:10.716586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:10.716437 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaaefd942a2572fa712041bfb66bb5382c6d0d0292e78e7343c1fa24371ee841" Oct 30 14:27:13.332018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.331980 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr"] Oct 30 14:27:13.332505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.332484 4968 cpu_manager.go:414] "containerMap: removing container" podUID="caf26b3d-b7e9-4003-90c9-a9ec050247ae" containerName="revoke-keys" Oct 30 14:27:13.332505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.332499 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf26b3d-b7e9-4003-90c9-a9ec050247ae" containerName="revoke-keys" Oct 30 14:27:13.332662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.332592 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf26b3d-b7e9-4003-90c9-a9ec050247ae" containerName="revoke-keys" Oct 30 14:27:13.336658 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.336639 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:13.351197 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.351175 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:27:13.354632 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.354609 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr"] Oct 30 14:27:13.359774 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.359755 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:27:13.359917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.359864 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:27:13.361161 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.361143 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:27:13.443542 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.443499 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35222d09-b538-4534-869f-286bcb99de9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba7ab5f-247fr\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:13.443686 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.443642 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35222d09-b538-4534-869f-286bcb99de9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba7ab5f-247fr\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:13.443795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.443700 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rsx9\" (UniqueName: \"kubernetes.io/projected/35222d09-b538-4534-869f-286bcb99de9d-kube-api-access-8rsx9\") pod \"maas-api-revoke-sub-4ba7ab5f-247fr\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:13.545195 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.545149 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35222d09-b538-4534-869f-286bcb99de9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba7ab5f-247fr\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:13.545399 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.545255 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35222d09-b538-4534-869f-286bcb99de9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba7ab5f-247fr\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:13.545399 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.545314 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rsx9\" (UniqueName: \"kubernetes.io/projected/35222d09-b538-4534-869f-286bcb99de9d-kube-api-access-8rsx9\") pod \"maas-api-revoke-sub-4ba7ab5f-247fr\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:13.545969 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.545940 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35222d09-b538-4534-869f-286bcb99de9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba7ab5f-247fr\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:13.548226 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.548187 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35222d09-b538-4534-869f-286bcb99de9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba7ab5f-247fr\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:13.581544 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.581512 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rsx9\" (UniqueName: \"kubernetes.io/projected/35222d09-b538-4534-869f-286bcb99de9d-kube-api-access-8rsx9\") pod \"maas-api-revoke-sub-4ba7ab5f-247fr\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:13.650620 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.650539 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:13.967752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:13.967710 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr"] Oct 30 14:27:13.968681 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:13.968658 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35222d09_b538_4534_869f_286bcb99de9d.slice/crio-6006fc6e19cccc0b5ab2ab0af1859555f6a75accc97ff246bf1b9e715b8d62e5 WatchSource:0}: Error finding container 6006fc6e19cccc0b5ab2ab0af1859555f6a75accc97ff246bf1b9e715b8d62e5: Status 404 returned error can't find the container with id 6006fc6e19cccc0b5ab2ab0af1859555f6a75accc97ff246bf1b9e715b8d62e5 Oct 30 14:27:14.357864 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:14.357832 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:27:14.752584 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:14.752541 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" event={"ID":"35222d09-b538-4534-869f-286bcb99de9d","Type":"ContainerStarted","Data":"4d52c81e9329fb7a93a978feef1c53a045478a92c0ac3a8f2b1dafe871a134eb"} Oct 30 14:27:14.752809 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:14.752612 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" event={"ID":"35222d09-b538-4534-869f-286bcb99de9d","Type":"ContainerStarted","Data":"6006fc6e19cccc0b5ab2ab0af1859555f6a75accc97ff246bf1b9e715b8d62e5"} Oct 30 14:27:14.799486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:14.799439 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" podStartSLOduration=1.799424991 podStartE2EDuration="1.799424991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:14.794615894 +0000 UTC m=+1230.420059269" watchObservedRunningTime="2026-10-30 14:27:14.799424991 +0000 UTC m=+1230.424868365" Oct 30 14:27:15.801525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:15.801462 4968 generic.go:356] "Generic (PLEG): container finished" podID="35222d09-b538-4534-869f-286bcb99de9d" containerID="4d52c81e9329fb7a93a978feef1c53a045478a92c0ac3a8f2b1dafe871a134eb" exitCode=0 Oct 30 14:27:15.802140 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:15.802113 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" event={"ID":"35222d09-b538-4534-869f-286bcb99de9d","Type":"ContainerDied","Data":"4d52c81e9329fb7a93a978feef1c53a045478a92c0ac3a8f2b1dafe871a134eb"} Oct 30 14:27:17.043749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.043706 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:17.134383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.134278 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247"] Oct 30 14:27:17.170660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.170626 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35222d09-b538-4534-869f-286bcb99de9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35222d09-b538-4534-869f-286bcb99de9d-maas-api-xks-ca\") pod \"35222d09-b538-4534-869f-286bcb99de9d\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " Oct 30 14:27:17.170849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.170668 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35222d09-b538-4534-869f-286bcb99de9d-kube-api-access-8rsx9\" (UniqueName: \"kubernetes.io/projected/35222d09-b538-4534-869f-286bcb99de9d-kube-api-access-8rsx9\") pod \"35222d09-b538-4534-869f-286bcb99de9d\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " Oct 30 14:27:17.170849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.170798 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35222d09-b538-4534-869f-286bcb99de9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35222d09-b538-4534-869f-286bcb99de9d-maas-api-openshift-ca\") pod \"35222d09-b538-4534-869f-286bcb99de9d\" (UID: \"35222d09-b538-4534-869f-286bcb99de9d\") " Oct 30 14:27:17.171235 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.171204 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35222d09-b538-4534-869f-286bcb99de9d-maas-api-openshift-ca" pod "35222d09-b538-4534-869f-286bcb99de9d" (UID: "35222d09-b538-4534-869f-286bcb99de9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:17.173002 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.172973 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35222d09-b538-4534-869f-286bcb99de9d-maas-api-xks-ca" pod "35222d09-b538-4534-869f-286bcb99de9d" (UID: "35222d09-b538-4534-869f-286bcb99de9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:17.173002 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.172996 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35222d09-b538-4534-869f-286bcb99de9d-kube-api-access-8rsx9" pod "35222d09-b538-4534-869f-286bcb99de9d" (UID: "35222d09-b538-4534-869f-286bcb99de9d"). InnerVolumeSpecName "kube-api-access-8rsx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:17.272707 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.272673 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35222d09-b538-4534-869f-286bcb99de9d-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:17.272707 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.272704 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rsx9\" (UniqueName: \"kubernetes.io/projected/35222d09-b538-4534-869f-286bcb99de9d-kube-api-access-8rsx9\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:17.272940 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.272742 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35222d09-b538-4534-869f-286bcb99de9d-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:17.863680 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.863645 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" Oct 30 14:27:17.863899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.863640 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr" event={"ID":"35222d09-b538-4534-869f-286bcb99de9d","Type":"ContainerDied","Data":"6006fc6e19cccc0b5ab2ab0af1859555f6a75accc97ff246bf1b9e715b8d62e5"} Oct 30 14:27:17.863899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:17.863773 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6006fc6e19cccc0b5ab2ab0af1859555f6a75accc97ff246bf1b9e715b8d62e5" Oct 30 14:27:18.544969 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.544930 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm"] Oct 30 14:27:18.545690 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.545668 4968 cpu_manager.go:414] "containerMap: removing container" podUID="35222d09-b538-4534-869f-286bcb99de9d" containerName="revoke-keys" Oct 30 14:27:18.545819 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.545692 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35222d09-b538-4534-869f-286bcb99de9d" containerName="revoke-keys" Oct 30 14:27:18.545881 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.545869 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="35222d09-b538-4534-869f-286bcb99de9d" containerName="revoke-keys" Oct 30 14:27:18.549519 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.549500 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:18.565519 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.565495 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff"] Oct 30 14:27:18.570074 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.570057 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:18.570594 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.570562 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:27:18.570594 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.570571 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:27:18.573780 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.573757 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:27:18.573903 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.573805 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:27:18.576335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.576311 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm"] Oct 30 14:27:18.598297 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.598257 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff"] Oct 30 14:27:18.692380 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.692348 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8x4\" (UniqueName: \"kubernetes.io/projected/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-kube-api-access-lb8x4\") pod \"maas-api-revoke-sub-85f48cb7-2gsmm\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:18.692380 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.692385 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242a80bc-bb8ff\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:18.692630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.692467 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242a80bc-bb8ff\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:18.692630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.692493 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f48cb7-2gsmm\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:18.692630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.692519 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhhxk\" (UniqueName: \"kubernetes.io/projected/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-kube-api-access-nhhxk\") pod \"maas-api-revoke-sub-242a80bc-bb8ff\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:18.692806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.692746 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f48cb7-2gsmm\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:18.793591 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.793558 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8x4\" (UniqueName: \"kubernetes.io/projected/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-kube-api-access-lb8x4\") pod \"maas-api-revoke-sub-85f48cb7-2gsmm\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:18.793591 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.793593 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242a80bc-bb8ff\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:18.793849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.793643 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242a80bc-bb8ff\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:18.793849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.793667 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f48cb7-2gsmm\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:18.793849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.793697 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhhxk\" (UniqueName: \"kubernetes.io/projected/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-kube-api-access-nhhxk\") pod \"maas-api-revoke-sub-242a80bc-bb8ff\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:18.793849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.793832 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f48cb7-2gsmm\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:18.794369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.794344 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f48cb7-2gsmm\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:18.794480 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.794418 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242a80bc-bb8ff\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:18.796191 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.796128 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242a80bc-bb8ff\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:18.796191 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.796184 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f48cb7-2gsmm\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:18.815543 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.815518 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhhxk\" (UniqueName: \"kubernetes.io/projected/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-kube-api-access-nhhxk\") pod \"maas-api-revoke-sub-242a80bc-bb8ff\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:18.818393 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.818374 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8x4\" (UniqueName: \"kubernetes.io/projected/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-kube-api-access-lb8x4\") pod \"maas-api-revoke-sub-85f48cb7-2gsmm\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:18.862845 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.862818 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:18.886297 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:18.886262 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:19.045610 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:19.045579 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5"] Oct 30 14:27:19.049831 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:19.049764 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8212e5-tpbc5"] Oct 30 14:27:19.185158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:19.185127 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm"] Oct 30 14:27:19.200055 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:19.200026 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69e8ee8_c3dd_4028_b3e8_e394d3ce0e9e.slice/crio-698435b09fce7d07979b42de1f4ee1e31804cbd06eb3d66d758ad44be96c3676 WatchSource:0}: Error finding container 698435b09fce7d07979b42de1f4ee1e31804cbd06eb3d66d758ad44be96c3676: Status 404 returned error can't find the container with id 698435b09fce7d07979b42de1f4ee1e31804cbd06eb3d66d758ad44be96c3676 Oct 30 14:27:19.219364 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:19.219338 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd61a99c_6d0b_4a82_935a_6fc3d14bacc6.slice/crio-bd4eb327284fc6ef047c70b7c6c681681dc82ea6a03ef715e04c8aa7a411c546 WatchSource:0}: Error finding container bd4eb327284fc6ef047c70b7c6c681681dc82ea6a03ef715e04c8aa7a411c546: Status 404 returned error can't find the container with id bd4eb327284fc6ef047c70b7c6c681681dc82ea6a03ef715e04c8aa7a411c546 Oct 30 14:27:19.219898 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:19.219873 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff"] Oct 30 14:27:19.880166 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:19.880127 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" event={"ID":"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6","Type":"ContainerStarted","Data":"6afd9fe95f41fad6d7334fda1d27f1573162e5c4462473f91db800d1688a0a29"} Oct 30 14:27:19.880637 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:19.880186 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" event={"ID":"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6","Type":"ContainerStarted","Data":"bd4eb327284fc6ef047c70b7c6c681681dc82ea6a03ef715e04c8aa7a411c546"} Oct 30 14:27:19.881948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:19.881921 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" event={"ID":"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e","Type":"ContainerStarted","Data":"6470ded6b65adfaf701312774d7ea580db10c9a842ef889a1a95834f6d98fce8"} Oct 30 14:27:19.882087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:19.881971 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" event={"ID":"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e","Type":"ContainerStarted","Data":"698435b09fce7d07979b42de1f4ee1e31804cbd06eb3d66d758ad44be96c3676"} Oct 30 14:27:19.920849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:19.920801 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" podStartSLOduration=1.9207864639999999 podStartE2EDuration="1.920786464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:19.916925118 +0000 UTC m=+1235.542368491" watchObservedRunningTime="2026-10-30 14:27:19.920786464 +0000 UTC m=+1235.546229866" Oct 30 14:27:19.959545 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:19.959490 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" podStartSLOduration=1.959476467 podStartE2EDuration="1.959476467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:19.955019328 +0000 UTC m=+1235.580462701" watchObservedRunningTime="2026-10-30 14:27:19.959476467 +0000 UTC m=+1235.584919839" Oct 30 14:27:20.446017 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:27:20.445932 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:27:20.518796 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:20.518762 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a502d9-143b-450e-9096-fc733d804245" path="/var/lib/kubelet/pods/e5a502d9-143b-450e-9096-fc733d804245/volumes" Oct 30 14:27:20.890486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:20.890449 4968 generic.go:356] "Generic (PLEG): container finished" podID="dd61a99c-6d0b-4a82-935a-6fc3d14bacc6" containerID="6afd9fe95f41fad6d7334fda1d27f1573162e5c4462473f91db800d1688a0a29" exitCode=0 Oct 30 14:27:20.890992 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:20.890538 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" event={"ID":"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6","Type":"ContainerDied","Data":"6afd9fe95f41fad6d7334fda1d27f1573162e5c4462473f91db800d1688a0a29"} Oct 30 14:27:20.892429 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:20.892403 4968 generic.go:356] "Generic (PLEG): container finished" podID="a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e" containerID="6470ded6b65adfaf701312774d7ea580db10c9a842ef889a1a95834f6d98fce8" exitCode=0 Oct 30 14:27:20.892541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:20.892489 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" event={"ID":"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e","Type":"ContainerDied","Data":"6470ded6b65adfaf701312774d7ea580db10c9a842ef889a1a95834f6d98fce8"} Oct 30 14:27:22.075284 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.075260 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:22.077319 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.077299 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:22.134772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.134736 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" podUID="8f24f5a6-883e-468e-bc08-4d9d47ec8864" containerName="payload-pre-processing" containerID="cri-o://b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a" gracePeriod=25 Oct 30 14:27:22.135058 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.135037 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-kube-api-access-lb8x4\" (UniqueName: \"kubernetes.io/projected/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-kube-api-access-lb8x4\") pod \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " Oct 30 14:27:22.135171 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.135074 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-xks-ca\") pod \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " Oct 30 14:27:22.135171 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.135128 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-kube-api-access-nhhxk\" (UniqueName: \"kubernetes.io/projected/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-kube-api-access-nhhxk\") pod \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " Oct 30 14:27:22.135385 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.135197 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-xks-ca\") pod \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " Oct 30 14:27:22.135385 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.135289 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-openshift-ca\") pod \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\" (UID: \"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6\") " Oct 30 14:27:22.135385 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.135349 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-openshift-ca\") pod \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\" (UID: \"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e\") " Oct 30 14:27:22.136118 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.136089 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-openshift-ca" pod "a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e" (UID: "a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:22.136219 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.136162 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-openshift-ca" pod "dd61a99c-6d0b-4a82-935a-6fc3d14bacc6" (UID: "dd61a99c-6d0b-4a82-935a-6fc3d14bacc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:22.137856 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.137827 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-xks-ca" pod "dd61a99c-6d0b-4a82-935a-6fc3d14bacc6" (UID: "dd61a99c-6d0b-4a82-935a-6fc3d14bacc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:22.138343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.138322 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-xks-ca" pod "a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e" (UID: "a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:22.138343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.138334 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-kube-api-access-lb8x4" pod "a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e" (UID: "a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e"). InnerVolumeSpecName "kube-api-access-lb8x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:22.145138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.145112 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-kube-api-access-nhhxk" pod "dd61a99c-6d0b-4a82-935a-6fc3d14bacc6" (UID: "dd61a99c-6d0b-4a82-935a-6fc3d14bacc6"). InnerVolumeSpecName "kube-api-access-nhhxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:22.236633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.236541 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhhxk\" (UniqueName: \"kubernetes.io/projected/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-kube-api-access-nhhxk\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:22.236633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.236585 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:22.236633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.236600 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:22.236633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.236614 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:22.236633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.236628 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb8x4\" (UniqueName: \"kubernetes.io/projected/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-kube-api-access-lb8x4\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:22.236929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.236641 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:22.912305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.912273 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" Oct 30 14:27:22.912305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.912305 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm" event={"ID":"a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e","Type":"ContainerDied","Data":"698435b09fce7d07979b42de1f4ee1e31804cbd06eb3d66d758ad44be96c3676"} Oct 30 14:27:22.912589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.912361 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="698435b09fce7d07979b42de1f4ee1e31804cbd06eb3d66d758ad44be96c3676" Oct 30 14:27:22.914108 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.914075 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" event={"ID":"dd61a99c-6d0b-4a82-935a-6fc3d14bacc6","Type":"ContainerDied","Data":"bd4eb327284fc6ef047c70b7c6c681681dc82ea6a03ef715e04c8aa7a411c546"} Oct 30 14:27:22.914108 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.914103 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff" Oct 30 14:27:22.914331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:22.914129 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd4eb327284fc6ef047c70b7c6c681681dc82ea6a03ef715e04c8aa7a411c546" Oct 30 14:27:24.035168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.035132 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g"] Oct 30 14:27:24.045503 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.045474 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47155609-89n7g"] Oct 30 14:27:24.413912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.413878 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92"] Oct 30 14:27:24.414461 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.414440 4968 cpu_manager.go:414] "containerMap: removing container" podUID="dd61a99c-6d0b-4a82-935a-6fc3d14bacc6" containerName="revoke-keys" Oct 30 14:27:24.414585 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.414460 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd61a99c-6d0b-4a82-935a-6fc3d14bacc6" containerName="revoke-keys" Oct 30 14:27:24.414585 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.414477 4968 cpu_manager.go:414] "containerMap: removing container" podUID="a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e" containerName="revoke-keys" Oct 30 14:27:24.414585 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.414485 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e" containerName="revoke-keys" Oct 30 14:27:24.414764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.414654 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd61a99c-6d0b-4a82-935a-6fc3d14bacc6" containerName="revoke-keys" Oct 30 14:27:24.414764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.414668 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e" containerName="revoke-keys" Oct 30 14:27:24.420744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.419903 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:24.438393 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.438367 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92"] Oct 30 14:27:24.442589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.442562 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:27:24.442698 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.442615 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:27:24.446548 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.446529 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:27:24.446650 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.446606 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:27:24.518651 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.518617 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6edd68-f51f-4e9c-b814-2af7b22c3b53" path="/var/lib/kubelet/pods/ea6edd68-f51f-4e9c-b814-2af7b22c3b53/volumes" Oct 30 14:27:24.573915 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.573882 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-137f01-48x92\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:24.574086 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.574066 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbv79\" (UniqueName: \"kubernetes.io/projected/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-kube-api-access-bbv79\") pod \"maas-api-revoke-keys-e2e-worker-w4-137f01-48x92\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:24.574130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.574112 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-137f01-48x92\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:24.675617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.675524 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-137f01-48x92\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:24.675798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.675651 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-137f01-48x92\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:24.675885 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.675823 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbv79\" (UniqueName: \"kubernetes.io/projected/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-kube-api-access-bbv79\") pod \"maas-api-revoke-keys-e2e-worker-w4-137f01-48x92\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:24.676311 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.676285 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-137f01-48x92\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:24.678269 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.678240 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-137f01-48x92\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:24.697916 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.697886 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbv79\" (UniqueName: \"kubernetes.io/projected/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-kube-api-access-bbv79\") pod \"maas-api-revoke-keys-e2e-worker-w4-137f01-48x92\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:24.738491 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:24.738470 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:25.038962 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:25.038871 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc"] Oct 30 14:27:25.045700 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:25.045666 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6009ef72-dtprc"] Oct 30 14:27:25.050129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:25.050103 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92"] Oct 30 14:27:25.051403 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:25.051373 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30dcc87_c0eb_4ef4_93a0_a6517a5fecbd.slice/crio-39c7284047b90f6716ebd1aece011290afad54c24a658fefb1841fd88fd291e6 WatchSource:0}: Error finding container 39c7284047b90f6716ebd1aece011290afad54c24a658fefb1841fd88fd291e6: Status 404 returned error can't find the container with id 39c7284047b90f6716ebd1aece011290afad54c24a658fefb1841fd88fd291e6 Oct 30 14:27:25.941746 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:25.939166 4968 generic.go:356] "Generic (PLEG): container finished" podID="f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd" containerID="bbc7e29cfefd573c6e7347c80d1823f21327e6047a4120d505977ac64e319b90" exitCode=0 Oct 30 14:27:25.941746 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:25.939295 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" event={"ID":"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd","Type":"ContainerDied","Data":"bbc7e29cfefd573c6e7347c80d1823f21327e6047a4120d505977ac64e319b90"} Oct 30 14:27:25.941746 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:25.939395 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" event={"ID":"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd","Type":"ContainerStarted","Data":"39c7284047b90f6716ebd1aece011290afad54c24a658fefb1841fd88fd291e6"} Oct 30 14:27:26.519871 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:26.519831 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f34edf8-b354-4392-ac1e-4a65dde459b0" path="/var/lib/kubelet/pods/8f34edf8-b354-4392-ac1e-4a65dde459b0/volumes" Oct 30 14:27:27.095743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.095707 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:27.218141 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.218051 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-openshift-ca\") pod \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " Oct 30 14:27:27.218141 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.218110 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-xks-ca\") pod \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " Oct 30 14:27:27.218385 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.218196 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-kube-api-access-bbv79\" (UniqueName: \"kubernetes.io/projected/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-kube-api-access-bbv79\") pod \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\" (UID: \"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd\") " Oct 30 14:27:27.218498 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.218472 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-openshift-ca" pod "f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd" (UID: "f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:27.218688 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.218666 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:27.220321 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.220288 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-kube-api-access-bbv79" pod "f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd" (UID: "f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd"). InnerVolumeSpecName "kube-api-access-bbv79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:27.220450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.220403 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-xks-ca" pod "f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd" (UID: "f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:27.319765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.319729 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbv79\" (UniqueName: \"kubernetes.io/projected/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-kube-api-access-bbv79\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:27.319765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.319763 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:27.956295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.956264 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" Oct 30 14:27:27.956736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.956309 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92" event={"ID":"f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd","Type":"ContainerDied","Data":"39c7284047b90f6716ebd1aece011290afad54c24a658fefb1841fd88fd291e6"} Oct 30 14:27:27.956736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:27.956360 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39c7284047b90f6716ebd1aece011290afad54c24a658fefb1841fd88fd291e6" Oct 30 14:27:28.033754 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:28.033707 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl"] Oct 30 14:27:28.038109 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:28.038085 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20be9e14-qftcl"] Oct 30 14:27:28.248202 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:28.248118 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" podUID="8f24f5a6-883e-468e-bc08-4d9d47ec8864" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:27:28.518587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:28.518497 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8b52a4-e6ca-4416-be2e-629351fc37f9" path="/var/lib/kubelet/pods/ae8b52a4-e6ca-4416-be2e-629351fc37f9/volumes" Oct 30 14:27:29.223111 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:29.223070 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv"] Oct 30 14:27:29.233183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:29.233151 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5"] Oct 30 14:27:29.240641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:29.240607 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj"] Oct 30 14:27:29.744086 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:29.744051 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp"] Oct 30 14:27:29.748198 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:29.748173 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-717995-25vsp"] Oct 30 14:27:30.032641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.032549 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57"] Oct 30 14:27:30.036630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.036602 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138eee7f-9rf57"] Oct 30 14:27:30.297577 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.297498 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn"] Oct 30 14:27:30.298026 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.298012 4968 cpu_manager.go:414] "containerMap: removing container" podUID="f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd" containerName="revoke-keys" Oct 30 14:27:30.298071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.298028 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd" containerName="revoke-keys" Oct 30 14:27:30.298135 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.298126 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd" containerName="revoke-keys" Oct 30 14:27:30.302433 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.302410 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:30.317271 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.317247 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:27:30.332567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.332539 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn"] Oct 30 14:27:30.332679 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.332639 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:27:30.332859 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.332835 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:27:30.332972 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.332884 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:27:30.358755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.358725 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86clm\" (UniqueName: \"kubernetes.io/projected/894b4b7c-3a69-4649-a000-f45598e52d4d-kube-api-access-86clm\") pod \"maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:30.358904 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.358823 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:30.358904 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.358869 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:30.460152 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.460116 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:30.460316 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.460254 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:30.460407 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.460390 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86clm\" (UniqueName: \"kubernetes.io/projected/894b4b7c-3a69-4649-a000-f45598e52d4d-kube-api-access-86clm\") pod \"maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:30.460817 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.460790 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:30.462707 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.462679 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:30.482387 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.482359 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86clm\" (UniqueName: \"kubernetes.io/projected/894b4b7c-3a69-4649-a000-f45598e52d4d-kube-api-access-86clm\") pod \"maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:30.520127 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.520094 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67cc2eb0-e6ac-412e-8618-a8ae5489c6a4" path="/var/lib/kubelet/pods/67cc2eb0-e6ac-412e-8618-a8ae5489c6a4/volumes" Oct 30 14:27:30.520527 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.520514 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf26b3d-b7e9-4003-90c9-a9ec050247ae" path="/var/lib/kubelet/pods/caf26b3d-b7e9-4003-90c9-a9ec050247ae/volumes" Oct 30 14:27:30.618142 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.618103 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:30.930644 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.930619 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn"] Oct 30 14:27:30.933324 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:30.933295 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894b4b7c_3a69_4649_a000_f45598e52d4d.slice/crio-89df4ee1e011095dc4519ade9599194c35ce2b936997c316f6f466c91f638398 WatchSource:0}: Error finding container 89df4ee1e011095dc4519ade9599194c35ce2b936997c316f6f466c91f638398: Status 404 returned error can't find the container with id 89df4ee1e011095dc4519ade9599194c35ce2b936997c316f6f466c91f638398 Oct 30 14:27:30.990016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:30.989976 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" event={"ID":"894b4b7c-3a69-4649-a000-f45598e52d4d","Type":"ContainerStarted","Data":"89df4ee1e011095dc4519ade9599194c35ce2b936997c316f6f466c91f638398"} Oct 30 14:27:31.998462 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:31.998382 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7"] Oct 30 14:27:32.003180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.003159 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.004292 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.004259 4968 generic.go:356] "Generic (PLEG): container finished" podID="894b4b7c-3a69-4649-a000-f45598e52d4d" containerID="3d0475ac6ef5e260d63b1357312f0fd9830cc368fec643d48cd8fcfb74fd52c9" exitCode=0 Oct 30 14:27:32.004406 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.004301 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" event={"ID":"894b4b7c-3a69-4649-a000-f45598e52d4d","Type":"ContainerDied","Data":"3d0475ac6ef5e260d63b1357312f0fd9830cc368fec643d48cd8fcfb74fd52c9"} Oct 30 14:27:32.027821 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.027784 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7"] Oct 30 14:27:32.077030 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.076991 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.077200 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.077047 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.077200 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.077163 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbpbp\" (UniqueName: \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-kube-api-access-fbpbp\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.077278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.077239 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.077315 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.077287 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.077356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.077331 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.077412 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.077399 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.077462 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.077450 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.077510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.077481 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.125954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.125920 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn"] Oct 30 14:27:32.130134 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.130113 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.160804 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.160774 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn"] Oct 30 14:27:32.162508 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.162485 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh"] Oct 30 14:27:32.166849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.166828 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.178546 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178522 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.178664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178561 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l8jl\" (UniqueName: \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-kube-api-access-7l8jl\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.178664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178597 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.178664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178623 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.178664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178659 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.178899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178692 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.178899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178742 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.178899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178769 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.178899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178835 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.178899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178868 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.179151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178907 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.179151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178959 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.179151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.178986 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c58c8\" (UniqueName: \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-kube-api-access-c58c8\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.179151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179018 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.179151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179052 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.179151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179082 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.179151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179121 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179155 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179181 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179223 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179282 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179343 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbpbp\" (UniqueName: \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-kube-api-access-fbpbp\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179368 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd4fc0f-9b38-4861-8869-03dc19722864-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179417 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179441 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179473 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179504 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.179983 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.180677 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.180965 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.181646 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.182334 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.184933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.182938 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.191023 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.190995 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh"] Oct 30 14:27:32.193066 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.192927 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.203648 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.203620 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.204953 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.204927 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbpbp\" (UniqueName: \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-kube-api-access-fbpbp\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.280094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280013 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.280094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280051 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.280094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280070 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.280322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280187 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.280322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280233 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.280322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280254 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.280474 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280343 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd4fc0f-9b38-4861-8869-03dc19722864-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.280474 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280387 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.280474 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280420 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.280474 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280449 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.280935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280484 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.280935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280503 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.280935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280513 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8jl\" (UniqueName: \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-kube-api-access-7l8jl\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.280935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280505 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.280935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280552 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.280935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280627 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.280935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280821 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.280935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280905 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.281356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280946 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.281356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.280972 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.281356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.281039 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.281356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.281107 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.281356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.281133 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c58c8\" (UniqueName: \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-kube-api-access-c58c8\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.281356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.281195 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-credential-socket\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.281672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.281439 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd4fc0f-9b38-4861-8869-03dc19722864-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.281750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.281674 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-certs\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.281750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.281698 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-data\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.282019 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.281999 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istiod-ca-cert\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.282905 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.282886 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.283058 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.283043 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.283434 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.283416 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-envoy\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.283476 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.283439 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-podinfo\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.304924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.304898 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.305081 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.305060 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-token\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.305220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.305202 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8jl\" (UniqueName: \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-kube-api-access-7l8jl\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.305428 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.305208 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c58c8\" (UniqueName: \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-kube-api-access-c58c8\") pod \"e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.318878 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.318856 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:32.444801 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.444758 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:32.501786 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.501360 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:32.650116 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.650080 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7"] Oct 30 14:27:32.650854 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:32.650829 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6abc3e1_e0d0_41f4_b180_f82afe240a07.slice/crio-915e98579684f550d2f6ab3f7719412bae38b3c3f3abd75f97020936243df135 WatchSource:0}: Error finding container 915e98579684f550d2f6ab3f7719412bae38b3c3f3abd75f97020936243df135: Status 404 returned error can't find the container with id 915e98579684f550d2f6ab3f7719412bae38b3c3f3abd75f97020936243df135 Oct 30 14:27:32.653433 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.653402 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:32.653494 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.653478 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:32.653526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.653509 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:32.754763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.754731 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn"] Oct 30 14:27:32.757993 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:32.757964 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd4fc0f_9b38_4861_8869_03dc19722864.slice/crio-feda7ebe93b9937837906766aaf76ed3a86eadcd6c90f89146a744dd8d014eef WatchSource:0}: Error finding container feda7ebe93b9937837906766aaf76ed3a86eadcd6c90f89146a744dd8d014eef: Status 404 returned error can't find the container with id feda7ebe93b9937837906766aaf76ed3a86eadcd6c90f89146a744dd8d014eef Oct 30 14:27:32.760644 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.760598 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:32.760776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.760747 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:32.760843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.760796 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:32.849786 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.849760 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh"] Oct 30 14:27:32.851986 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:32.851947 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode294e23e_573e_4f62_a2b8_27a3dd0009b7.slice/crio-812c5fb5522ec987ca09ab1b04f17afeab2aca93d3882c0080bd7553b63e421b WatchSource:0}: Error finding container 812c5fb5522ec987ca09ab1b04f17afeab2aca93d3882c0080bd7553b63e421b: Status 404 returned error can't find the container with id 812c5fb5522ec987ca09ab1b04f17afeab2aca93d3882c0080bd7553b63e421b Oct 30 14:27:32.854405 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.854374 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:32.854488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.854437 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:32.854488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:32.854467 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:33.022365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.022256 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" event={"ID":"e294e23e-573e-4f62-a2b8-27a3dd0009b7","Type":"ContainerStarted","Data":"c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38"} Oct 30 14:27:33.022365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.022317 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" event={"ID":"e294e23e-573e-4f62-a2b8-27a3dd0009b7","Type":"ContainerStarted","Data":"812c5fb5522ec987ca09ab1b04f17afeab2aca93d3882c0080bd7553b63e421b"} Oct 30 14:27:33.024183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.023617 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" event={"ID":"a6abc3e1-e0d0-41f4-b180-f82afe240a07","Type":"ContainerStarted","Data":"9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a"} Oct 30 14:27:33.024183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.023673 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" event={"ID":"a6abc3e1-e0d0-41f4-b180-f82afe240a07","Type":"ContainerStarted","Data":"915e98579684f550d2f6ab3f7719412bae38b3c3f3abd75f97020936243df135"} Oct 30 14:27:33.026883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.026837 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" event={"ID":"3dd4fc0f-9b38-4861-8869-03dc19722864","Type":"ContainerStarted","Data":"872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3"} Oct 30 14:27:33.027012 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.026890 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" event={"ID":"3dd4fc0f-9b38-4861-8869-03dc19722864","Type":"ContainerStarted","Data":"feda7ebe93b9937837906766aaf76ed3a86eadcd6c90f89146a744dd8d014eef"} Oct 30 14:27:33.105735 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.088113 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podStartSLOduration=1.088093656 podStartE2EDuration="1.088093656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:33.079138652 +0000 UTC m=+1248.704582027" watchObservedRunningTime="2026-10-30 14:27:33.088093656 +0000 UTC m=+1248.713537031" Oct 30 14:27:33.136811 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.136321 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" podStartSLOduration=2.136304197 podStartE2EDuration="2.136304197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:33.121959577 +0000 UTC m=+1248.747402952" watchObservedRunningTime="2026-10-30 14:27:33.136304197 +0000 UTC m=+1248.761747570" Oct 30 14:27:33.173398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.173108 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podStartSLOduration=1.173087709 podStartE2EDuration="1.173087709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:33.166636365 +0000 UTC m=+1248.792079739" watchObservedRunningTime="2026-10-30 14:27:33.173087709 +0000 UTC m=+1248.798531083" Oct 30 14:27:33.248250 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.248222 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:33.303304 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.303264 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894b4b7c-3a69-4649-a000-f45598e52d4d-kube-api-access-86clm\" (UniqueName: \"kubernetes.io/projected/894b4b7c-3a69-4649-a000-f45598e52d4d-kube-api-access-86clm\") pod \"894b4b7c-3a69-4649-a000-f45598e52d4d\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " Oct 30 14:27:33.303510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.303489 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-openshift-ca\") pod \"894b4b7c-3a69-4649-a000-f45598e52d4d\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " Oct 30 14:27:33.304042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.303541 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-xks-ca\") pod \"894b4b7c-3a69-4649-a000-f45598e52d4d\" (UID: \"894b4b7c-3a69-4649-a000-f45598e52d4d\") " Oct 30 14:27:33.304460 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.304013 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-openshift-ca" pod "894b4b7c-3a69-4649-a000-f45598e52d4d" (UID: "894b4b7c-3a69-4649-a000-f45598e52d4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:33.304565 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.304506 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:33.311084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.311055 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894b4b7c-3a69-4649-a000-f45598e52d4d-kube-api-access-86clm" pod "894b4b7c-3a69-4649-a000-f45598e52d4d" (UID: "894b4b7c-3a69-4649-a000-f45598e52d4d"). InnerVolumeSpecName "kube-api-access-86clm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:33.311351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.311322 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-xks-ca" pod "894b4b7c-3a69-4649-a000-f45598e52d4d" (UID: "894b4b7c-3a69-4649-a000-f45598e52d4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:33.319738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.319700 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:33.321268 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.321242 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 14:27:33.321379 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.321347 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 14:27:33.407293 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.407208 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86clm\" (UniqueName: \"kubernetes.io/projected/894b4b7c-3a69-4649-a000-f45598e52d4d-kube-api-access-86clm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:33.407293 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.407247 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894b4b7c-3a69-4649-a000-f45598e52d4d-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:33.445210 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.445180 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:33.446703 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.446681 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 14:27:33.446789 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.446748 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 14:27:33.506907 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.506874 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:33.509435 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.509365 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 14:27:33.509677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:33.509479 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 14:27:34.074887 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:34.074844 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" event={"ID":"894b4b7c-3a69-4649-a000-f45598e52d4d","Type":"ContainerDied","Data":"89df4ee1e011095dc4519ade9599194c35ce2b936997c316f6f466c91f638398"} Oct 30 14:27:34.075439 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:34.074921 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89df4ee1e011095dc4519ade9599194c35ce2b936997c316f6f466c91f638398" Oct 30 14:27:34.075439 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:34.074864 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn" Oct 30 14:27:34.337764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:34.333474 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 14:27:34.337764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:34.333536 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 14:27:34.451740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:34.448196 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 14:27:34.451740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:34.448261 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 14:27:34.502738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:34.502275 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 14:27:34.502738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:34.502351 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 14:27:35.136790 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.135768 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq"] Oct 30 14:27:35.151643 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.151611 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks"] Oct 30 14:27:35.390948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.327112 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 14:27:35.390948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.327177 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 14:27:35.390948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.348844 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs"] Oct 30 14:27:35.390948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.349652 4968 cpu_manager.go:414] "containerMap: removing container" podUID="894b4b7c-3a69-4649-a000-f45598e52d4d" containerName="revoke-keys" Oct 30 14:27:35.390948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.349672 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894b4b7c-3a69-4649-a000-f45598e52d4d" containerName="revoke-keys" Oct 30 14:27:35.390948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.350215 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="894b4b7c-3a69-4649-a000-f45598e52d4d" containerName="revoke-keys" Oct 30 14:27:35.390948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.354614 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:35.511740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.425433 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:27:35.511740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.425515 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:27:35.511740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.425580 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:27:35.511740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.425665 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:27:35.511740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.427016 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4csz\" (UniqueName: \"kubernetes.io/projected/84e15b9e-ff32-4a61-a41b-d02514acd47c-kube-api-access-m4csz\") pod \"maas-api-revoke-sub-8c9f7c83-9xdqs\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:35.511740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.427098 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c9f7c83-9xdqs\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:35.511740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.427165 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c9f7c83-9xdqs\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:35.511740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.456602 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 14:27:35.511740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.456656 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 14:27:35.639865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.547989 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c9f7c83-9xdqs\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:35.639865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.548081 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c9f7c83-9xdqs\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:35.639865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.548231 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4csz\" (UniqueName: \"kubernetes.io/projected/84e15b9e-ff32-4a61-a41b-d02514acd47c-kube-api-access-m4csz\") pod \"maas-api-revoke-sub-8c9f7c83-9xdqs\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:35.639865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.572111 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c9f7c83-9xdqs\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:35.639865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.572299 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs"] Oct 30 14:27:35.639865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.625845 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 14:27:35.639865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.625902 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 14:27:35.749423 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.748955 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4csz\" (UniqueName: \"kubernetes.io/projected/84e15b9e-ff32-4a61-a41b-d02514acd47c-kube-api-access-m4csz\") pod \"maas-api-revoke-sub-8c9f7c83-9xdqs\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:35.925009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:35.924965 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs"] Oct 30 14:27:36.025124 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.017251 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c9f7c83-9xdqs\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:36.025124 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.023352 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:36.159764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.152841 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" podUID="c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:27:36.170753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.160785 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs"] Oct 30 14:27:36.170753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.163285 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7075a8bf-q4fxs\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:36.170753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.163471 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw8mc\" (UniqueName: \"kubernetes.io/projected/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-kube-api-access-tw8mc\") pod \"maas-api-revoke-sub-7075a8bf-q4fxs\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:36.170753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.163595 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7075a8bf-q4fxs\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:36.278251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.278163 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:36.375096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.375054 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw8mc\" (UniqueName: \"kubernetes.io/projected/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-kube-api-access-tw8mc\") pod \"maas-api-revoke-sub-7075a8bf-q4fxs\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:36.375260 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.375194 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7075a8bf-q4fxs\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:36.375321 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.375269 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7075a8bf-q4fxs\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:36.389200 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.389166 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7075a8bf-q4fxs\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:36.398390 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.398357 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7075a8bf-q4fxs\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:36.425219 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.424770 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 14:27:36.425219 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.424826 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 14:27:36.425429 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.425406 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw8mc\" (UniqueName: \"kubernetes.io/projected/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-kube-api-access-tw8mc\") pod \"maas-api-revoke-sub-7075a8bf-q4fxs\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:36.468153 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.468118 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 14:27:36.468408 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.468380 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 14:27:36.487221 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.487188 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:36.507219 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.507174 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 14:27:36.507529 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.507494 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 14:27:36.822021 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:36.821985 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:27:37.369088 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:37.369055 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 14:27:37.371407 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:37.369690 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 14:27:37.511755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:37.469784 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 14:27:37.511755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:37.469842 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 14:27:37.511755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:37.477178 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs"] Oct 30 14:27:37.564752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:37.562403 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 14:27:37.564752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:37.562488 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 14:27:37.867749 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:37.849281 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab82e4e2_5ac1_4904_a0ee_c9b724547a0f.slice/crio-7a8d9ee07e7cbaeec0b724af2149d547d3b088324816f8274973b90d83c58a95 WatchSource:0}: Error finding container 7a8d9ee07e7cbaeec0b724af2149d547d3b088324816f8274973b90d83c58a95: Status 404 returned error can't find the container with id 7a8d9ee07e7cbaeec0b724af2149d547d3b088324816f8274973b90d83c58a95 Oct 30 14:27:37.875039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:37.874181 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs"] Oct 30 14:27:38.099345 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.094354 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg"] Oct 30 14:27:38.130740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.129994 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e9f918-kb2qg"] Oct 30 14:27:38.308916 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.308823 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" podUID="8f24f5a6-883e-468e-bc08-4d9d47ec8864" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:27:38.324350 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.324319 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 14:27:38.324605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.324578 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 14:27:38.448503 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.448474 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 14:27:38.449087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.449050 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 14:27:38.523776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.523699 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 14:27:38.524705 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.524672 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 14:27:38.529823 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.529793 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56cb76e2-0b34-4d52-9297-abab9c9c53ef" path="/var/lib/kubelet/pods/56cb76e2-0b34-4d52-9297-abab9c9c53ef/volumes" Oct 30 14:27:38.654818 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.654780 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" event={"ID":"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f","Type":"ContainerStarted","Data":"ea9a5c6e44253daae09b4d17e9fe7c7c3424eb4145bc73d561d011a27df0ffea"} Oct 30 14:27:38.655070 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.655048 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" event={"ID":"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f","Type":"ContainerStarted","Data":"7a8d9ee07e7cbaeec0b724af2149d547d3b088324816f8274973b90d83c58a95"} Oct 30 14:27:38.666854 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.666819 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" event={"ID":"84e15b9e-ff32-4a61-a41b-d02514acd47c","Type":"ContainerStarted","Data":"a5bdf7b7c19383bf9ff3566620a32ce50b3ab18aa1096c9fd4f4701d6ed4f96e"} Oct 30 14:27:38.667100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.667078 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" event={"ID":"84e15b9e-ff32-4a61-a41b-d02514acd47c","Type":"ContainerStarted","Data":"4ef7119963197f92efade1ca729907f793a4609596762daa666e4f89e4a26532"} Oct 30 14:27:38.868966 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.868915 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws"] Oct 30 14:27:38.869456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:38.869419 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" podUID="92a2483e-29a2-4369-8d98-3e0860468bb0" containerName="istio-proxy" containerID="cri-o://b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2" gracePeriod=30 Oct 30 14:27:39.036737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.003385 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" podStartSLOduration=4.003361399 podStartE2EDuration="4.003361399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:38.726450499 +0000 UTC m=+1254.351893873" watchObservedRunningTime="2026-10-30 14:27:39.003361399 +0000 UTC m=+1254.628804774" Oct 30 14:27:39.074744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.028787 4968 patch_prober.go:29] interesting pod/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:27:39.074744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.047263 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" podUID="92a2483e-29a2-4369-8d98-3e0860468bb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:27:39.115741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.107976 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" podStartSLOduration=4.107952515 podStartE2EDuration="4.107952515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:38.785396766 +0000 UTC m=+1254.410840148" watchObservedRunningTime="2026-10-30 14:27:39.107952515 +0000 UTC m=+1254.733395888" Oct 30 14:27:39.148888 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.148819 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz"] Oct 30 14:27:39.152177 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.152112 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91920ce-dgnkz"] Oct 30 14:27:39.336750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.333024 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 14:27:39.336750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.333104 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 14:27:39.454742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.452799 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 14:27:39.454742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.452859 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 14:27:39.506981 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.506231 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 14:27:39.506981 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.506308 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 14:27:39.768699 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.766743 4968 generic.go:356] "Generic (PLEG): container finished" podID="ab82e4e2-5ac1-4904-a0ee-c9b724547a0f" containerID="ea9a5c6e44253daae09b4d17e9fe7c7c3424eb4145bc73d561d011a27df0ffea" exitCode=0 Oct 30 14:27:39.768699 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.768281 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" event={"ID":"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f","Type":"ContainerDied","Data":"ea9a5c6e44253daae09b4d17e9fe7c7c3424eb4145bc73d561d011a27df0ffea"} Oct 30 14:27:39.771382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.771353 4968 generic.go:356] "Generic (PLEG): container finished" podID="84e15b9e-ff32-4a61-a41b-d02514acd47c" containerID="a5bdf7b7c19383bf9ff3566620a32ce50b3ab18aa1096c9fd4f4701d6ed4f96e" exitCode=0 Oct 30 14:27:39.771517 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:39.771465 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" event={"ID":"84e15b9e-ff32-4a61-a41b-d02514acd47c","Type":"ContainerDied","Data":"a5bdf7b7c19383bf9ff3566620a32ce50b3ab18aa1096c9fd4f4701d6ed4f96e"} Oct 30 14:27:40.320016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:40.319982 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 14:27:40.320244 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:40.320047 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 14:27:40.445397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:40.445354 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 14:27:40.445597 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:40.445421 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 14:27:40.502782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:40.502702 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 14:27:40.503280 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:40.502813 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 14:27:40.524298 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:40.524251 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0373cdd-31d6-464c-8baf-c8f422f6a00b" path="/var/lib/kubelet/pods/b0373cdd-31d6-464c-8baf-c8f422f6a00b/volumes" Oct 30 14:27:40.803824 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:40.803773 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4"] Oct 30 14:27:41.349792 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.342050 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:41.477007 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.476929 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:41.583742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.554925 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-xks-ca\") pod \"84e15b9e-ff32-4a61-a41b-d02514acd47c\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " Oct 30 14:27:41.593001 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.592972 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-openshift-ca\") pod \"84e15b9e-ff32-4a61-a41b-d02514acd47c\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " Oct 30 14:27:41.597039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.597012 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-kube-api-access-tw8mc\" (UniqueName: \"kubernetes.io/projected/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-kube-api-access-tw8mc\") pod \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " Oct 30 14:27:41.597667 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.597646 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-openshift-ca\") pod \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " Oct 30 14:27:41.602067 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.602046 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e15b9e-ff32-4a61-a41b-d02514acd47c-kube-api-access-m4csz\" (UniqueName: \"kubernetes.io/projected/84e15b9e-ff32-4a61-a41b-d02514acd47c-kube-api-access-m4csz\") pod \"84e15b9e-ff32-4a61-a41b-d02514acd47c\" (UID: \"84e15b9e-ff32-4a61-a41b-d02514acd47c\") " Oct 30 14:27:41.602564 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.602544 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-xks-ca\") pod \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\" (UID: \"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f\") " Oct 30 14:27:41.612731 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.596920 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-openshift-ca" pod "84e15b9e-ff32-4a61-a41b-d02514acd47c" (UID: "84e15b9e-ff32-4a61-a41b-d02514acd47c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:41.641095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.601912 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-openshift-ca" pod "ab82e4e2-5ac1-4904-a0ee-c9b724547a0f" (UID: "ab82e4e2-5ac1-4904-a0ee-c9b724547a0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:41.764022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.711806 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:41.764022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.711845 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:41.764022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.762070 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-kube-api-access-tw8mc" pod "ab82e4e2-5ac1-4904-a0ee-c9b724547a0f" (UID: "ab82e4e2-5ac1-4904-a0ee-c9b724547a0f"). InnerVolumeSpecName "kube-api-access-tw8mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:41.764022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.762206 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-xks-ca" pod "ab82e4e2-5ac1-4904-a0ee-c9b724547a0f" (UID: "ab82e4e2-5ac1-4904-a0ee-c9b724547a0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:41.764022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.762323 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e15b9e-ff32-4a61-a41b-d02514acd47c-kube-api-access-m4csz" pod "84e15b9e-ff32-4a61-a41b-d02514acd47c" (UID: "84e15b9e-ff32-4a61-a41b-d02514acd47c"). InnerVolumeSpecName "kube-api-access-m4csz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:41.794895 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.794840 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-xks-ca" pod "84e15b9e-ff32-4a61-a41b-d02514acd47c" (UID: "84e15b9e-ff32-4a61-a41b-d02514acd47c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:41.824794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.821686 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw8mc\" (UniqueName: \"kubernetes.io/projected/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-kube-api-access-tw8mc\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:41.824794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.821739 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4csz\" (UniqueName: \"kubernetes.io/projected/84e15b9e-ff32-4a61-a41b-d02514acd47c-kube-api-access-m4csz\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:41.824794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.821761 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:41.824794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.821777 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e15b9e-ff32-4a61-a41b-d02514acd47c-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:41.846671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.846620 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" event={"ID":"84e15b9e-ff32-4a61-a41b-d02514acd47c","Type":"ContainerDied","Data":"4ef7119963197f92efade1ca729907f793a4609596762daa666e4f89e4a26532"} Oct 30 14:27:41.846865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.846694 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ef7119963197f92efade1ca729907f793a4609596762daa666e4f89e4a26532" Oct 30 14:27:41.846865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.846795 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs" Oct 30 14:27:41.855106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.853202 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" event={"ID":"ab82e4e2-5ac1-4904-a0ee-c9b724547a0f","Type":"ContainerDied","Data":"7a8d9ee07e7cbaeec0b724af2149d547d3b088324816f8274973b90d83c58a95"} Oct 30 14:27:41.855106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.853256 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a8d9ee07e7cbaeec0b724af2149d547d3b088324816f8274973b90d83c58a95" Oct 30 14:27:41.855106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.853319 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs" Oct 30 14:27:41.955792 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.955764 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:41.957826 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.957799 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:41.958733 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:41.958690 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:27:42.053845 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.053683 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n"] Oct 30 14:27:42.073193 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.073109 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77051e7e-r4q5n"] Oct 30 14:27:42.448322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.448284 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:27:42.448542 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.448508 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:27:42.482270 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.482239 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:27:42.504541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.504459 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:27:42.504769 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.504524 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:27:42.527038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.526218 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189c32fe-20bf-45e5-9aba-154f23d5e895" path="/var/lib/kubelet/pods/189c32fe-20bf-45e5-9aba-154f23d5e895/volumes" Oct 30 14:27:42.538974 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.538927 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f24f5a6-883e-468e-bc08-4d9d47ec8864-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f24f5a6-883e-468e-bc08-4d9d47ec8864-plugins-config-volume\") pod \"8f24f5a6-883e-468e-bc08-4d9d47ec8864\" (UID: \"8f24f5a6-883e-468e-bc08-4d9d47ec8864\") " Oct 30 14:27:42.539158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.539016 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f24f5a6-883e-468e-bc08-4d9d47ec8864-kube-api-access-c8b94\" (UniqueName: \"kubernetes.io/projected/8f24f5a6-883e-468e-bc08-4d9d47ec8864-kube-api-access-c8b94\") pod \"8f24f5a6-883e-468e-bc08-4d9d47ec8864\" (UID: \"8f24f5a6-883e-468e-bc08-4d9d47ec8864\") " Oct 30 14:27:42.539366 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.539338 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f24f5a6-883e-468e-bc08-4d9d47ec8864-plugins-config-volume" pod "8f24f5a6-883e-468e-bc08-4d9d47ec8864" (UID: "8f24f5a6-883e-468e-bc08-4d9d47ec8864"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:42.539574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.539552 4968 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f24f5a6-883e-468e-bc08-4d9d47ec8864-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:42.544084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.544052 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f24f5a6-883e-468e-bc08-4d9d47ec8864-kube-api-access-c8b94" pod "8f24f5a6-883e-468e-bc08-4d9d47ec8864" (UID: "8f24f5a6-883e-468e-bc08-4d9d47ec8864"). InnerVolumeSpecName "kube-api-access-c8b94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:42.641335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.641249 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8b94\" (UniqueName: \"kubernetes.io/projected/8f24f5a6-883e-468e-bc08-4d9d47ec8864-kube-api-access-c8b94\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:42.864566 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.864527 4968 generic.go:356] "Generic (PLEG): container finished" podID="8f24f5a6-883e-468e-bc08-4d9d47ec8864" containerID="b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a" exitCode=0 Oct 30 14:27:42.865661 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.865640 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" Oct 30 14:27:42.866872 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.866842 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" event={"ID":"8f24f5a6-883e-468e-bc08-4d9d47ec8864","Type":"ContainerDied","Data":"b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a"} Oct 30 14:27:42.867098 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.867062 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247" event={"ID":"8f24f5a6-883e-468e-bc08-4d9d47ec8864","Type":"ContainerDied","Data":"ec4bf08c7cf32ca34d21acf0e19a7f4157deaf0e2d8b21051f6e184e9c58b7ce"} Oct 30 14:27:42.867250 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.867192 4968 scope.go:122] "RemoveContainer" containerID="b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a" Oct 30 14:27:42.891951 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.889219 4968 scope.go:122] "RemoveContainer" containerID="b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a" Oct 30 14:27:42.899751 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:27:42.895991 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a\": container with ID starting with b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a not found: ID does not exist" containerID="b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a" Oct 30 14:27:42.899751 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.896036 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a"} err="failed to get container status \"b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a\": rpc error: code = NotFound desc = could not find container \"b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a\": container with ID starting with b064198447c03ae7cdc967c27ccf560d9503c42fb1ce51d1623316507d85a16a not found: ID does not exist" Oct 30 14:27:42.960993 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.960896 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247"] Oct 30 14:27:42.965752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:42.964658 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-717995-64b9677747-5z247"] Oct 30 14:27:43.058741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.052768 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m"] Oct 30 14:27:43.070737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.070692 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd222ee-fw64m"] Oct 30 14:27:43.514676 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.514641 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:27:43.514988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.514959 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:27:43.549917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.549876 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:27:43.550188 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.550156 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:27:43.669536 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.669387 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx"] Oct 30 14:27:43.673562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.670206 4968 cpu_manager.go:414] "containerMap: removing container" podUID="8f24f5a6-883e-468e-bc08-4d9d47ec8864" containerName="payload-pre-processing" Oct 30 14:27:43.673562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.670226 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f24f5a6-883e-468e-bc08-4d9d47ec8864" containerName="payload-pre-processing" Oct 30 14:27:43.673562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.670241 4968 cpu_manager.go:414] "containerMap: removing container" podUID="84e15b9e-ff32-4a61-a41b-d02514acd47c" containerName="revoke-keys" Oct 30 14:27:43.673562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.670246 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e15b9e-ff32-4a61-a41b-d02514acd47c" containerName="revoke-keys" Oct 30 14:27:43.673562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.670256 4968 cpu_manager.go:414] "containerMap: removing container" podUID="ab82e4e2-5ac1-4904-a0ee-c9b724547a0f" containerName="revoke-keys" Oct 30 14:27:43.673562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.670266 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab82e4e2-5ac1-4904-a0ee-c9b724547a0f" containerName="revoke-keys" Oct 30 14:27:43.673562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.670382 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f24f5a6-883e-468e-bc08-4d9d47ec8864" containerName="payload-pre-processing" Oct 30 14:27:43.673562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.670391 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e15b9e-ff32-4a61-a41b-d02514acd47c" containerName="revoke-keys" Oct 30 14:27:43.673562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.670398 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab82e4e2-5ac1-4904-a0ee-c9b724547a0f" containerName="revoke-keys" Oct 30 14:27:43.679734 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.677114 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:43.692729 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.691852 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:27:43.692729 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.691985 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:27:43.701182 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.700830 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx"] Oct 30 14:27:43.704034 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.703051 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:27:43.704034 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.703141 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:27:43.729116 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.728361 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:43.796961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.796498 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:43.796961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.796894 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx"] Oct 30 14:27:43.808058 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.807800 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx"] Oct 30 14:27:43.837593 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.828527 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:43.846160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.845631 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dpxj\" (UniqueName: \"kubernetes.io/projected/d3240528-e402-43de-9702-45c2c308588f-kube-api-access-9dpxj\") pod \"maas-api-revoke-sub-2d9de6fc-qjbsx\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:43.846160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.845702 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpddf\" (UniqueName: \"kubernetes.io/projected/35fa2371-a482-470e-b019-baa5cd14b25c-kube-api-access-vpddf\") pod \"maas-api-revoke-sub-9a41b3f8-lwsjx\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:43.846160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.845765 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3240528-e402-43de-9702-45c2c308588f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9de6fc-qjbsx\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:43.846160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.845842 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a41b3f8-lwsjx\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:43.846160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.845956 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3240528-e402-43de-9702-45c2c308588f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9de6fc-qjbsx\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:43.846160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.845986 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a41b3f8-lwsjx\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:43.871531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.871494 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:27:43.947743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.947600 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpddf\" (UniqueName: \"kubernetes.io/projected/35fa2371-a482-470e-b019-baa5cd14b25c-kube-api-access-vpddf\") pod \"maas-api-revoke-sub-9a41b3f8-lwsjx\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:43.947743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.947637 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3240528-e402-43de-9702-45c2c308588f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9de6fc-qjbsx\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:43.947743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.947699 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a41b3f8-lwsjx\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:43.948052 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.947796 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3240528-e402-43de-9702-45c2c308588f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9de6fc-qjbsx\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:43.948052 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.947817 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a41b3f8-lwsjx\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:43.948052 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.947898 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dpxj\" (UniqueName: \"kubernetes.io/projected/d3240528-e402-43de-9702-45c2c308588f-kube-api-access-9dpxj\") pod \"maas-api-revoke-sub-2d9de6fc-qjbsx\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:43.949142 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.948944 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a41b3f8-lwsjx\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:43.949586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.949561 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3240528-e402-43de-9702-45c2c308588f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9de6fc-qjbsx\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:43.951385 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.951354 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3240528-e402-43de-9702-45c2c308588f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9de6fc-qjbsx\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:43.953749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.953709 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a41b3f8-lwsjx\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:43.970412 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.970385 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dpxj\" (UniqueName: \"kubernetes.io/projected/d3240528-e402-43de-9702-45c2c308588f-kube-api-access-9dpxj\") pod \"maas-api-revoke-sub-2d9de6fc-qjbsx\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:43.970560 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:43.970508 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpddf\" (UniqueName: \"kubernetes.io/projected/35fa2371-a482-470e-b019-baa5cd14b25c-kube-api-access-vpddf\") pod \"maas-api-revoke-sub-9a41b3f8-lwsjx\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:44.022174 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.022139 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:44.022656 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.022626 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:44.023453 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.023427 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:27:44.124953 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.124921 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:44.140955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.140838 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:44.210481 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.210452 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:27:44.255142 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.255103 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" podUID="726af6c4-5d67-47dc-8d3a-abbaec756c58" containerName="main" containerID="cri-o://43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e" gracePeriod=45 Oct 30 14:27:44.257051 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.256885 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" podUID="bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" containerName="main" containerID="cri-o://52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2" gracePeriod=45 Oct 30 14:27:44.270397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.269425 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" podUID="c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" containerName="main" containerID="cri-o://b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201" gracePeriod=45 Oct 30 14:27:44.305652 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.305615 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" podUID="bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": dial tcp 10.132.0.121:8000: connect: connection refused" Oct 30 14:27:44.355332 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.355304 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-podinfo\") pod \"92a2483e-29a2-4369-8d98-3e0860468bb0\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " Oct 30 14:27:44.355502 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.355404 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-envoy\") pod \"92a2483e-29a2-4369-8d98-3e0860468bb0\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " Oct 30 14:27:44.355564 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.355496 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-data\") pod \"92a2483e-29a2-4369-8d98-3e0860468bb0\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " Oct 30 14:27:44.355564 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.355536 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-credential-socket\") pod \"92a2483e-29a2-4369-8d98-3e0860468bb0\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " Oct 30 14:27:44.355671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.355563 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-kube-api-access-f4xq5\" (UniqueName: \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-kube-api-access-f4xq5\") pod \"92a2483e-29a2-4369-8d98-3e0860468bb0\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " Oct 30 14:27:44.355671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.355651 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-socket\") pod \"92a2483e-29a2-4369-8d98-3e0860468bb0\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " Oct 30 14:27:44.355814 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.355750 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-token\" (UniqueName: \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-token\") pod \"92a2483e-29a2-4369-8d98-3e0860468bb0\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " Oct 30 14:27:44.355814 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.355804 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a2483e-29a2-4369-8d98-3e0860468bb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a2483e-29a2-4369-8d98-3e0860468bb0-istiod-ca-cert\") pod \"92a2483e-29a2-4369-8d98-3e0860468bb0\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " Oct 30 14:27:44.355931 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.355865 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-certs\") pod \"92a2483e-29a2-4369-8d98-3e0860468bb0\" (UID: \"92a2483e-29a2-4369-8d98-3e0860468bb0\") " Oct 30 14:27:44.356427 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.355969 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-credential-socket" pod "92a2483e-29a2-4369-8d98-3e0860468bb0" (UID: "92a2483e-29a2-4369-8d98-3e0860468bb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.356427 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.356306 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-data" pod "92a2483e-29a2-4369-8d98-3e0860468bb0" (UID: "92a2483e-29a2-4369-8d98-3e0860468bb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.356427 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.356368 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.357104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.356535 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-certs" pod "92a2483e-29a2-4369-8d98-3e0860468bb0" (UID: "92a2483e-29a2-4369-8d98-3e0860468bb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.357104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.356830 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-socket" pod "92a2483e-29a2-4369-8d98-3e0860468bb0" (UID: "92a2483e-29a2-4369-8d98-3e0860468bb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.357104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.356931 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a2483e-29a2-4369-8d98-3e0860468bb0-istiod-ca-cert" pod "92a2483e-29a2-4369-8d98-3e0860468bb0" (UID: "92a2483e-29a2-4369-8d98-3e0860468bb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:44.358179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.358124 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-podinfo" pod "92a2483e-29a2-4369-8d98-3e0860468bb0" (UID: "92a2483e-29a2-4369-8d98-3e0860468bb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:27:44.358276 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.358249 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-envoy" pod "92a2483e-29a2-4369-8d98-3e0860468bb0" (UID: "92a2483e-29a2-4369-8d98-3e0860468bb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.358513 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.358483 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-token" pod "92a2483e-29a2-4369-8d98-3e0860468bb0" (UID: "92a2483e-29a2-4369-8d98-3e0860468bb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:44.359879 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.359847 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-kube-api-access-f4xq5" pod "92a2483e-29a2-4369-8d98-3e0860468bb0" (UID: "92a2483e-29a2-4369-8d98-3e0860468bb0"). InnerVolumeSpecName "kube-api-access-f4xq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:44.458363 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.458050 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a2483e-29a2-4369-8d98-3e0860468bb0-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.458363 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.458087 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.458363 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.458102 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.458363 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.458114 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.458363 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.458130 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.458363 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.458143 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xq5\" (UniqueName: \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-kube-api-access-f4xq5\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.458363 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.458156 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2483e-29a2-4369-8d98-3e0860468bb0-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.458363 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.458170 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a2483e-29a2-4369-8d98-3e0860468bb0-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.485635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.484998 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx"] Oct 30 14:27:44.487400 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:44.487371 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3240528_e402_43de_9702_45c2c308588f.slice/crio-18ca457da13f2a11f56f2a28a915978667bd00f779fb4e5a4a67e19ff8b68273 WatchSource:0}: Error finding container 18ca457da13f2a11f56f2a28a915978667bd00f779fb4e5a4a67e19ff8b68273: Status 404 returned error can't find the container with id 18ca457da13f2a11f56f2a28a915978667bd00f779fb4e5a4a67e19ff8b68273 Oct 30 14:27:44.499846 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.498959 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx"] Oct 30 14:27:44.507503 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:27:44.507466 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fa2371_a482_470e_b019_baa5cd14b25c.slice/crio-deb2c7df1edeb350c46f2426bbdd6349766e9d9ef3225052b8308631b4a3e7f1 WatchSource:0}: Error finding container deb2c7df1edeb350c46f2426bbdd6349766e9d9ef3225052b8308631b4a3e7f1: Status 404 returned error can't find the container with id deb2c7df1edeb350c46f2426bbdd6349766e9d9ef3225052b8308631b4a3e7f1 Oct 30 14:27:44.530925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.530895 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308ea75f-c712-4435-a0b7-5134ff7bd65c" path="/var/lib/kubelet/pods/308ea75f-c712-4435-a0b7-5134ff7bd65c/volumes" Oct 30 14:27:44.531779 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.531617 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f24f5a6-883e-468e-bc08-4d9d47ec8864" path="/var/lib/kubelet/pods/8f24f5a6-883e-468e-bc08-4d9d47ec8864/volumes" Oct 30 14:27:44.552406 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.552305 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:27:44.659812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.659781 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-kube-api-access-5npzq\" (UniqueName: \"kubernetes.io/projected/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-kube-api-access-5npzq\") pod \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " Oct 30 14:27:44.659975 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.659950 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tls-certs\") pod \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " Oct 30 14:27:44.660057 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.659994 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-dshm\") pod \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " Oct 30 14:27:44.660057 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.660032 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tmp-dir\") pod \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " Oct 30 14:27:44.660163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.660084 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-home\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-home\") pod \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " Oct 30 14:27:44.660163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.660137 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-model-cache\") pod \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\" (UID: \"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908\") " Oct 30 14:27:44.660472 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.660343 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tmp-dir" pod "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" (UID: "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.660472 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.660368 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-home" pod "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" (UID: "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.660676 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.660564 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-model-cache" pod "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" (UID: "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.660836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.660809 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.660836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.660836 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.661044 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.660850 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.679850 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.679820 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-dshm" pod "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" (UID: "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.680228 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.679877 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-kube-api-access-5npzq" pod "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" (UID: "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908"). InnerVolumeSpecName "kube-api-access-5npzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:44.680562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.680524 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tls-certs" pod "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" (UID: "bc1c7d01-3c70-43e8-bdb6-6fb97b60b908"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:44.712666 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.712641 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:27:44.716205 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.716182 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:27:44.762189 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.762157 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5npzq\" (UniqueName: \"kubernetes.io/projected/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-kube-api-access-5npzq\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.762189 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.762188 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.762189 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.762203 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.863053 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863019 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-home\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-home\") pod \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " Oct 30 14:27:44.863246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863088 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tls-certs\") pod \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " Oct 30 14:27:44.863246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863131 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-home\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-home\") pod \"726af6c4-5d67-47dc-8d3a-abbaec756c58\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " Oct 30 14:27:44.863246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863163 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tmp-dir\") pod \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " Oct 30 14:27:44.863246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863190 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-model-cache\") pod \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " Oct 30 14:27:44.863473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863249 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-kube-api-access-9wzlg\" (UniqueName: \"kubernetes.io/projected/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-kube-api-access-9wzlg\") pod \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " Oct 30 14:27:44.863473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863305 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-dshm\") pod \"726af6c4-5d67-47dc-8d3a-abbaec756c58\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " Oct 30 14:27:44.863473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863327 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-home" pod "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" (UID: "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.863473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863358 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-model-cache\") pod \"726af6c4-5d67-47dc-8d3a-abbaec756c58\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " Oct 30 14:27:44.863473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863392 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tmp-dir" pod "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" (UID: "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.863473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863404 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-home" pod "726af6c4-5d67-47dc-8d3a-abbaec756c58" (UID: "726af6c4-5d67-47dc-8d3a-abbaec756c58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.863473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863430 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-tmp-dir\") pod \"726af6c4-5d67-47dc-8d3a-abbaec756c58\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " Oct 30 14:27:44.863875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863488 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/726af6c4-5d67-47dc-8d3a-abbaec756c58-tls-certs\" (UniqueName: \"kubernetes.io/secret/726af6c4-5d67-47dc-8d3a-abbaec756c58-tls-certs\") pod \"726af6c4-5d67-47dc-8d3a-abbaec756c58\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " Oct 30 14:27:44.863875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863552 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-dshm\") pod \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\" (UID: \"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa\") " Oct 30 14:27:44.863875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863597 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726af6c4-5d67-47dc-8d3a-abbaec756c58-kube-api-access-468vr\" (UniqueName: \"kubernetes.io/projected/726af6c4-5d67-47dc-8d3a-abbaec756c58-kube-api-access-468vr\") pod \"726af6c4-5d67-47dc-8d3a-abbaec756c58\" (UID: \"726af6c4-5d67-47dc-8d3a-abbaec756c58\") " Oct 30 14:27:44.863875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863689 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-tmp-dir" pod "726af6c4-5d67-47dc-8d3a-abbaec756c58" (UID: "726af6c4-5d67-47dc-8d3a-abbaec756c58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.863875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.863751 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-model-cache" pod "726af6c4-5d67-47dc-8d3a-abbaec756c58" (UID: "726af6c4-5d67-47dc-8d3a-abbaec756c58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.864138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.864090 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.864138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.864111 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.864138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.864125 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.864138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.864137 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.864352 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.864149 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.864352 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.864275 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-model-cache" pod "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" (UID: "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.865819 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.865786 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-dshm" pod "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" (UID: "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.866096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.866072 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-kube-api-access-9wzlg" pod "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" (UID: "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa"). InnerVolumeSpecName "kube-api-access-9wzlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:44.866333 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.866303 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tls-certs" pod "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" (UID: "c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:44.866420 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.866336 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726af6c4-5d67-47dc-8d3a-abbaec756c58-tls-certs" pod "726af6c4-5d67-47dc-8d3a-abbaec756c58" (UID: "726af6c4-5d67-47dc-8d3a-abbaec756c58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:44.866420 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.866341 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-dshm" pod "726af6c4-5d67-47dc-8d3a-abbaec756c58" (UID: "726af6c4-5d67-47dc-8d3a-abbaec756c58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:44.866556 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.866538 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726af6c4-5d67-47dc-8d3a-abbaec756c58-kube-api-access-468vr" pod "726af6c4-5d67-47dc-8d3a-abbaec756c58" (UID: "726af6c4-5d67-47dc-8d3a-abbaec756c58"). InnerVolumeSpecName "kube-api-access-468vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:44.892769 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.892739 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" event={"ID":"35fa2371-a482-470e-b019-baa5cd14b25c","Type":"ContainerStarted","Data":"deb2c7df1edeb350c46f2426bbdd6349766e9d9ef3225052b8308631b4a3e7f1"} Oct 30 14:27:44.894472 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.894442 4968 generic.go:356] "Generic (PLEG): container finished" podID="bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" containerID="52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2" exitCode=0 Oct 30 14:27:44.894589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.894492 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" Oct 30 14:27:44.894589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.894524 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" event={"ID":"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908","Type":"ContainerDied","Data":"52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2"} Oct 30 14:27:44.894589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.894571 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5" event={"ID":"bc1c7d01-3c70-43e8-bdb6-6fb97b60b908","Type":"ContainerDied","Data":"e575d1aecc6010daeb25087ecd6ba6abc58381a887a4f88eaad299e22b361a36"} Oct 30 14:27:44.894780 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.894594 4968 scope.go:122] "RemoveContainer" containerID="52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2" Oct 30 14:27:44.904867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.897410 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" event={"ID":"d3240528-e402-43de-9702-45c2c308588f","Type":"ContainerStarted","Data":"81edb5d3490f01cbc97d8f3646bee263c0371c9c08c78a07929c88e0a9a438d4"} Oct 30 14:27:44.904867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.897671 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" event={"ID":"d3240528-e402-43de-9702-45c2c308588f","Type":"ContainerStarted","Data":"18ca457da13f2a11f56f2a28a915978667bd00f779fb4e5a4a67e19ff8b68273"} Oct 30 14:27:44.904867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.899942 4968 generic.go:356] "Generic (PLEG): container finished" podID="c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" containerID="b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201" exitCode=0 Oct 30 14:27:44.904867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.900077 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" event={"ID":"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa","Type":"ContainerDied","Data":"b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201"} Oct 30 14:27:44.904867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.900123 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" event={"ID":"c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa","Type":"ContainerDied","Data":"ea7ee8d61bf50d12d5fbe3f10617880607e0b624b46003610c8e6476501326a4"} Oct 30 14:27:44.904867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.900210 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj" Oct 30 14:27:44.911933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.910838 4968 generic.go:356] "Generic (PLEG): container finished" podID="726af6c4-5d67-47dc-8d3a-abbaec756c58" containerID="43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e" exitCode=0 Oct 30 14:27:44.911933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.910932 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" Oct 30 14:27:44.911933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.911021 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" event={"ID":"726af6c4-5d67-47dc-8d3a-abbaec756c58","Type":"ContainerDied","Data":"43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e"} Oct 30 14:27:44.911933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.911057 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" event={"ID":"726af6c4-5d67-47dc-8d3a-abbaec756c58","Type":"ContainerDied","Data":"b7f157a1a58c28914b3045898f157656c2a5a5953e69011905da11c0e16a8a38"} Oct 30 14:27:44.917994 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.917914 4968 generic.go:356] "Generic (PLEG): container finished" podID="92a2483e-29a2-4369-8d98-3e0860468bb0" containerID="b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2" exitCode=0 Oct 30 14:27:44.917994 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.917964 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" Oct 30 14:27:44.918243 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.918007 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" event={"ID":"92a2483e-29a2-4369-8d98-3e0860468bb0","Type":"ContainerDied","Data":"b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2"} Oct 30 14:27:44.918243 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.918083 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws" event={"ID":"92a2483e-29a2-4369-8d98-3e0860468bb0","Type":"ContainerDied","Data":"96f9efeda017c0224da8755b934064e92bc0888f9f82d0488b9274b486d7deeb"} Oct 30 14:27:44.920684 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.920665 4968 scope.go:122] "RemoveContainer" containerID="52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2" Oct 30 14:27:44.920974 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:27:44.920951 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2\": container with ID starting with 52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2 not found: ID does not exist" containerID="52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2" Oct 30 14:27:44.921089 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.920984 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2"} err="failed to get container status \"52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2\": rpc error: code = NotFound desc = could not find container \"52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2\": container with ID starting with 52f27eb8f3a1ba39f18928fa041adf55efc1b657fbc1d57168ffe55c3cd338d2 not found: ID does not exist" Oct 30 14:27:44.921089 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.921010 4968 scope.go:122] "RemoveContainer" containerID="b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201" Oct 30 14:27:44.933575 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.933554 4968 scope.go:122] "RemoveContainer" containerID="b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201" Oct 30 14:27:44.933909 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:27:44.933890 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201\": container with ID starting with b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201 not found: ID does not exist" containerID="b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201" Oct 30 14:27:44.933960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.933919 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201"} err="failed to get container status \"b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201\": rpc error: code = NotFound desc = could not find container \"b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201\": container with ID starting with b00a4f72123d5c9f01c8c9bf76bef95e823c091457719559166ba21f60af6201 not found: ID does not exist" Oct 30 14:27:44.933960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.933937 4968 scope.go:122] "RemoveContainer" containerID="43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e" Oct 30 14:27:44.943510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.943487 4968 scope.go:122] "RemoveContainer" containerID="43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e" Oct 30 14:27:44.943823 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:27:44.943803 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e\": container with ID starting with 43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e not found: ID does not exist" containerID="43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e" Oct 30 14:27:44.943894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.943833 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e"} err="failed to get container status \"43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e\": rpc error: code = NotFound desc = could not find container \"43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e\": container with ID starting with 43503321df9e4de87e44e8655dcbed9f775eac22606c10e42d1cbd4fb59d507e not found: ID does not exist" Oct 30 14:27:44.943894 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.943863 4968 scope.go:122] "RemoveContainer" containerID="b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2" Oct 30 14:27:44.954543 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.954520 4968 scope.go:122] "RemoveContainer" containerID="b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2" Oct 30 14:27:44.954853 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:27:44.954827 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2\": container with ID starting with b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2 not found: ID does not exist" containerID="b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2" Oct 30 14:27:44.954934 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.954866 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2"} err="failed to get container status \"b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2\": rpc error: code = NotFound desc = could not find container \"b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2\": container with ID starting with b1065fa08ae8e8b2fda4d482a67c2f1dbab111146c33cef408d9b8d209575bb2 not found: ID does not exist" Oct 30 14:27:44.962598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.962556 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" podStartSLOduration=1.962543162 podStartE2EDuration="1.962543162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:44.959223826 +0000 UTC m=+1260.584667201" watchObservedRunningTime="2026-10-30 14:27:44.962543162 +0000 UTC m=+1260.587986534" Oct 30 14:27:44.964742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.964706 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-468vr\" (UniqueName: \"kubernetes.io/projected/726af6c4-5d67-47dc-8d3a-abbaec756c58-kube-api-access-468vr\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.964742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.964744 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.964865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.964754 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.964865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.964763 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wzlg\" (UniqueName: \"kubernetes.io/projected/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-kube-api-access-9wzlg\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.964865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.964771 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/726af6c4-5d67-47dc-8d3a-abbaec756c58-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.964865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.964778 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/726af6c4-5d67-47dc-8d3a-abbaec756c58-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.964865 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.964787 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:44.995929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:44.995899 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj"] Oct 30 14:27:45.000041 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.000018 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/premium-w4-137f01-kserve-65c5f9cd5b-r5tmj"] Oct 30 14:27:45.030538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.030516 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5"] Oct 30 14:27:45.036572 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.036544 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb"] Oct 30 14:27:45.041245 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.041217 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-w4-137f01-kserve-7ddcfc56dc-x4xk5"] Oct 30 14:27:45.048152 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.047503 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c22439-xvkcb"] Oct 30 14:27:45.073243 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.073175 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws"] Oct 30 14:27:45.077783 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.077759 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-717995-openshift-default-5f475f5df5-bhgws"] Oct 30 14:27:45.106844 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.106815 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv"] Oct 30 14:27:45.111124 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.111100 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv"] Oct 30 14:27:45.675694 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.675588 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-137f01/distinct-2-w4-137f01-kserve-65d96fff96-dfcqv" podUID="726af6c4-5d67-47dc-8d3a-abbaec756c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:27:45.805972 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.805928 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" podUID="8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6" containerName="payload-pre-processing" containerID="cri-o://c230a8f34fc1f286942a0e74048b0cd1341d3ec9c5e45972d919ca0b1efd7934" gracePeriod=25 Oct 30 14:27:45.929423 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.929328 4968 generic.go:356] "Generic (PLEG): container finished" podID="d3240528-e402-43de-9702-45c2c308588f" containerID="81edb5d3490f01cbc97d8f3646bee263c0371c9c08c78a07929c88e0a9a438d4" exitCode=0 Oct 30 14:27:45.929575 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.929411 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" event={"ID":"d3240528-e402-43de-9702-45c2c308588f","Type":"ContainerDied","Data":"81edb5d3490f01cbc97d8f3646bee263c0371c9c08c78a07929c88e0a9a438d4"} Oct 30 14:27:45.931467 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.931440 4968 generic.go:356] "Generic (PLEG): container finished" podID="35fa2371-a482-470e-b019-baa5cd14b25c" containerID="7bed2748b202e4f5922ba68807a743457741e26643f45c2186af7982e3fb5221" exitCode=0 Oct 30 14:27:45.931616 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:45.931474 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" event={"ID":"35fa2371-a482-470e-b019-baa5cd14b25c","Type":"ContainerDied","Data":"7bed2748b202e4f5922ba68807a743457741e26643f45c2186af7982e3fb5221"} Oct 30 14:27:46.518890 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:46.518855 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726af6c4-5d67-47dc-8d3a-abbaec756c58" path="/var/lib/kubelet/pods/726af6c4-5d67-47dc-8d3a-abbaec756c58/volumes" Oct 30 14:27:46.519339 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:46.519325 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a2483e-29a2-4369-8d98-3e0860468bb0" path="/var/lib/kubelet/pods/92a2483e-29a2-4369-8d98-3e0860468bb0/volumes" Oct 30 14:27:46.519704 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:46.519691 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" path="/var/lib/kubelet/pods/bc1c7d01-3c70-43e8-bdb6-6fb97b60b908/volumes" Oct 30 14:27:46.520424 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:46.520408 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" path="/var/lib/kubelet/pods/c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa/volumes" Oct 30 14:27:46.520815 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:46.520803 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3653da-8836-46e6-8c3b-dff86258aa8a" path="/var/lib/kubelet/pods/fd3653da-8836-46e6-8c3b-dff86258aa8a/volumes" Oct 30 14:27:47.055870 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.055303 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7"] Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.055876 4968 cpu_manager.go:414] "containerMap: removing container" podUID="bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" containerName="main" Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.055890 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" containerName="main" Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.055902 4968 cpu_manager.go:414] "containerMap: removing container" podUID="92a2483e-29a2-4369-8d98-3e0860468bb0" containerName="istio-proxy" Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.055907 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a2483e-29a2-4369-8d98-3e0860468bb0" containerName="istio-proxy" Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.055923 4968 cpu_manager.go:414] "containerMap: removing container" podUID="c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" containerName="main" Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.055928 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" containerName="main" Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.055938 4968 cpu_manager.go:414] "containerMap: removing container" podUID="726af6c4-5d67-47dc-8d3a-abbaec756c58" containerName="main" Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.055943 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726af6c4-5d67-47dc-8d3a-abbaec756c58" containerName="main" Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.056043 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1c7d01-3c70-43e8-bdb6-6fb97b60b908" containerName="main" Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.056050 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a2483e-29a2-4369-8d98-3e0860468bb0" containerName="istio-proxy" Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.056060 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1dd55df-e0b0-4ffe-a221-96fd8ff4b3fa" containerName="main" Oct 30 14:27:47.056312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.056066 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="726af6c4-5d67-47dc-8d3a-abbaec756c58" containerName="main" Oct 30 14:27:47.067180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.067156 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.093799 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.092203 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7"] Oct 30 14:27:47.155075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.155051 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:47.159237 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.159217 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:47.197430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.196363 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.197430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.196416 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxbc\" (UniqueName: \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-kube-api-access-nnxbc\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.197430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.196544 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.197430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.196575 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.197430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.196604 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.197430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.196628 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.197430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.196676 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047af57e-0edc-42c0-bb08-e5306dddce4a-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.197430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.196755 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.197430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.196834 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.298068 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298027 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3240528-e402-43de-9702-45c2c308588f-kube-api-access-9dpxj\" (UniqueName: \"kubernetes.io/projected/d3240528-e402-43de-9702-45c2c308588f-kube-api-access-9dpxj\") pod \"d3240528-e402-43de-9702-45c2c308588f\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " Oct 30 14:27:47.298271 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298087 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-xks-ca\") pod \"35fa2371-a482-470e-b019-baa5cd14b25c\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " Oct 30 14:27:47.298351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298286 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-openshift-ca\") pod \"35fa2371-a482-470e-b019-baa5cd14b25c\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " Oct 30 14:27:47.298351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298325 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fa2371-a482-470e-b019-baa5cd14b25c-kube-api-access-vpddf\" (UniqueName: \"kubernetes.io/projected/35fa2371-a482-470e-b019-baa5cd14b25c-kube-api-access-vpddf\") pod \"35fa2371-a482-470e-b019-baa5cd14b25c\" (UID: \"35fa2371-a482-470e-b019-baa5cd14b25c\") " Oct 30 14:27:47.298456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298364 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3240528-e402-43de-9702-45c2c308588f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3240528-e402-43de-9702-45c2c308588f-maas-api-openshift-ca\") pod \"d3240528-e402-43de-9702-45c2c308588f\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " Oct 30 14:27:47.298456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298451 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3240528-e402-43de-9702-45c2c308588f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3240528-e402-43de-9702-45c2c308588f-maas-api-xks-ca\") pod \"d3240528-e402-43de-9702-45c2c308588f\" (UID: \"d3240528-e402-43de-9702-45c2c308588f\") " Oct 30 14:27:47.298667 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298646 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047af57e-0edc-42c0-bb08-e5306dddce4a-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.298778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298691 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-openshift-ca" pod "35fa2371-a482-470e-b019-baa5cd14b25c" (UID: "35fa2371-a482-470e-b019-baa5cd14b25c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:47.298778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298707 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.299048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298777 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3240528-e402-43de-9702-45c2c308588f-maas-api-openshift-ca" pod "d3240528-e402-43de-9702-45c2c308588f" (UID: "d3240528-e402-43de-9702-45c2c308588f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:47.299048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298846 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.299048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298905 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.299048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.298942 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxbc\" (UniqueName: \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-kube-api-access-nnxbc\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.299278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.299097 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.299278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.299129 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.299278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.299159 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.299278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.299185 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.299278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.299264 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:47.299529 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.299283 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3240528-e402-43de-9702-45c2c308588f-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:47.299529 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.299401 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047af57e-0edc-42c0-bb08-e5306dddce4a-istiod-ca-cert\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.299637 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.299608 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.300160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.300059 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-data\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.300261 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.300227 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-credential-socket\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.300807 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.300786 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-certs\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.301014 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.300980 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-xks-ca" pod "35fa2371-a482-470e-b019-baa5cd14b25c" (UID: "35fa2371-a482-470e-b019-baa5cd14b25c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:47.302161 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.302121 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fa2371-a482-470e-b019-baa5cd14b25c-kube-api-access-vpddf" pod "35fa2371-a482-470e-b019-baa5cd14b25c" (UID: "35fa2371-a482-470e-b019-baa5cd14b25c"). InnerVolumeSpecName "kube-api-access-vpddf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:47.302272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.302230 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3240528-e402-43de-9702-45c2c308588f-kube-api-access-9dpxj" pod "d3240528-e402-43de-9702-45c2c308588f" (UID: "d3240528-e402-43de-9702-45c2c308588f"). InnerVolumeSpecName "kube-api-access-9dpxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:47.302272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.302256 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3240528-e402-43de-9702-45c2c308588f-maas-api-xks-ca" pod "d3240528-e402-43de-9702-45c2c308588f" (UID: "d3240528-e402-43de-9702-45c2c308588f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:47.302432 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.302412 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-podinfo\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.302932 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.302907 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-envoy\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.321751 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.321710 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxbc\" (UniqueName: \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-kube-api-access-nnxbc\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.322027 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.322009 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-token\") pod \"e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.385389 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.385344 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:47.400278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.400242 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpddf\" (UniqueName: \"kubernetes.io/projected/35fa2371-a482-470e-b019-baa5cd14b25c-kube-api-access-vpddf\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:47.400278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.400276 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3240528-e402-43de-9702-45c2c308588f-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:47.400278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.400286 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dpxj\" (UniqueName: \"kubernetes.io/projected/d3240528-e402-43de-9702-45c2c308588f-kube-api-access-9dpxj\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:47.400486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.400295 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35fa2371-a482-470e-b019-baa5cd14b25c-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:47.731780 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.731746 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7"] Oct 30 14:27:47.737206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.737165 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:47.737317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.737242 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:47.737317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.737276 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:47.949419 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.949388 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" Oct 30 14:27:47.949419 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.949405 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx" event={"ID":"35fa2371-a482-470e-b019-baa5cd14b25c","Type":"ContainerDied","Data":"deb2c7df1edeb350c46f2426bbdd6349766e9d9ef3225052b8308631b4a3e7f1"} Oct 30 14:27:47.949641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.949459 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deb2c7df1edeb350c46f2426bbdd6349766e9d9ef3225052b8308631b4a3e7f1" Oct 30 14:27:47.951380 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.951350 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" event={"ID":"047af57e-0edc-42c0-bb08-e5306dddce4a","Type":"ContainerStarted","Data":"0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be"} Oct 30 14:27:47.951548 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.951396 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" event={"ID":"047af57e-0edc-42c0-bb08-e5306dddce4a","Type":"ContainerStarted","Data":"969e7670383e3d1c47a59d89ee36e281e6fe44b00852c55ac82e5d3430ce45bd"} Oct 30 14:27:47.952925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.952889 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" event={"ID":"d3240528-e402-43de-9702-45c2c308588f","Type":"ContainerDied","Data":"18ca457da13f2a11f56f2a28a915978667bd00f779fb4e5a4a67e19ff8b68273"} Oct 30 14:27:47.952925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.952911 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx" Oct 30 14:27:47.953127 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:47.952935 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18ca457da13f2a11f56f2a28a915978667bd00f779fb4e5a4a67e19ff8b68273" Oct 30 14:27:48.001447 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:48.001324 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" podStartSLOduration=1.00130375 podStartE2EDuration="1.00130375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:47.997137837 +0000 UTC m=+1263.622581214" watchObservedRunningTime="2026-10-30 14:27:48.00130375 +0000 UTC m=+1263.626747123" Oct 30 14:27:48.204737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:48.204186 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:27:48.391739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:48.390958 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:48.396772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:48.393866 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 14:27:48.396772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:48.393923 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 14:27:49.521740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:49.506878 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 14:27:49.521740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:49.506961 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 14:27:49.562216 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:49.560762 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92"] Oct 30 14:27:49.568792 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:49.568750 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-137f01-48x92"] Oct 30 14:27:50.275054 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:50.275008 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" podUID="aca3c382-c905-4be5-ae1a-d5f9336adc1d" containerName="main" containerID="cri-o://4a4ece1581879a3f8fe49c45f1de7155206b1d1d099a5294aad8502d4eda042e" gracePeriod=45 Oct 30 14:27:50.388139 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:50.388084 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 14:27:50.388420 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:50.388384 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 14:27:50.584907 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:50.584859 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd" path="/var/lib/kubelet/pods/f30dcc87-c0eb-4ef4-93a0-a6517a5fecbd/volumes" Oct 30 14:27:50.709803 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:27:50.703136 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:27:50.955416 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:50.955346 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" podUID="aca3c382-c905-4be5-ae1a-d5f9336adc1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": dial tcp 10.132.0.61:8000: connect: connection refused" Oct 30 14:27:51.667740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:51.622383 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" podUID="d9555cbc-5986-436f-8d2e-4550cbe5e49c" containerName="main" containerID="cri-o://47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4" gracePeriod=44 Oct 30 14:27:51.667740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:51.635630 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 14:27:51.667740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:51.635705 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 14:27:51.784729 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:51.756013 4968 generic.go:356] "Generic (PLEG): container finished" podID="aca3c382-c905-4be5-ae1a-d5f9336adc1d" containerID="4a4ece1581879a3f8fe49c45f1de7155206b1d1d099a5294aad8502d4eda042e" exitCode=0 Oct 30 14:27:51.784729 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:51.756200 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" event={"ID":"aca3c382-c905-4be5-ae1a-d5f9336adc1d","Type":"ContainerDied","Data":"4a4ece1581879a3f8fe49c45f1de7155206b1d1d099a5294aad8502d4eda042e"} Oct 30 14:27:51.850741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:51.826148 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" podUID="d9555cbc-5986-436f-8d2e-4550cbe5e49c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": EOF" Oct 30 14:27:51.890737 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:51.890078 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:27:52.087161 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.087114 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-dshm\") pod \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " Oct 30 14:27:52.087422 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.087407 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-home\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-home\") pod \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " Oct 30 14:27:52.087600 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.087575 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tmp-dir\") pod \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " Oct 30 14:27:52.106817 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.090925 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-model-cache\") pod \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " Oct 30 14:27:52.114106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.114075 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca3c382-c905-4be5-ae1a-d5f9336adc1d-kube-api-access-l2558\" (UniqueName: \"kubernetes.io/projected/aca3c382-c905-4be5-ae1a-d5f9336adc1d-kube-api-access-l2558\") pod \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " Oct 30 14:27:52.124906 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.124864 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-home" pod "aca3c382-c905-4be5-ae1a-d5f9336adc1d" (UID: "aca3c382-c905-4be5-ae1a-d5f9336adc1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:52.125210 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.125186 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-model-cache" pod "aca3c382-c905-4be5-ae1a-d5f9336adc1d" (UID: "aca3c382-c905-4be5-ae1a-d5f9336adc1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:52.125325 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.125014 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tmp-dir" pod "aca3c382-c905-4be5-ae1a-d5f9336adc1d" (UID: "aca3c382-c905-4be5-ae1a-d5f9336adc1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:52.125407 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.125126 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tls-certs\") pod \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\" (UID: \"aca3c382-c905-4be5-ae1a-d5f9336adc1d\") " Oct 30 14:27:52.126393 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.126374 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.126545 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.126531 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.126662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.126651 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.127830 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.127797 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-dshm" pod "aca3c382-c905-4be5-ae1a-d5f9336adc1d" (UID: "aca3c382-c905-4be5-ae1a-d5f9336adc1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:52.130512 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.130125 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca3c382-c905-4be5-ae1a-d5f9336adc1d-kube-api-access-l2558" pod "aca3c382-c905-4be5-ae1a-d5f9336adc1d" (UID: "aca3c382-c905-4be5-ae1a-d5f9336adc1d"). InnerVolumeSpecName "kube-api-access-l2558". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:52.145325 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.144663 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tls-certs" pod "aca3c382-c905-4be5-ae1a-d5f9336adc1d" (UID: "aca3c382-c905-4be5-ae1a-d5f9336adc1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:52.145325 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.144770 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" podUID="d9555cbc-5986-436f-8d2e-4550cbe5e49c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": dial tcp 10.132.0.119:8000: connect: connection refused" Oct 30 14:27:52.145325 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.144884 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:27:52.228433 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.228301 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aca3c382-c905-4be5-ae1a-d5f9336adc1d-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.228433 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.228334 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2558\" (UniqueName: \"kubernetes.io/projected/aca3c382-c905-4be5-ae1a-d5f9336adc1d-kube-api-access-l2558\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.228433 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.228346 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca3c382-c905-4be5-ae1a-d5f9336adc1d-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.270327 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.270296 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:27:52.329744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.328879 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-dshm\") pod \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " Oct 30 14:27:52.329744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.329220 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tmp-dir\") pod \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " Oct 30 14:27:52.329744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.329420 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tmp-dir" pod "d9555cbc-5986-436f-8d2e-4550cbe5e49c" (UID: "d9555cbc-5986-436f-8d2e-4550cbe5e49c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:52.329744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.329485 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tls-certs\") pod \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " Oct 30 14:27:52.329744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.329751 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-home\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-home\") pod \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " Oct 30 14:27:52.330159 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.329948 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-home" pod "d9555cbc-5986-436f-8d2e-4550cbe5e49c" (UID: "d9555cbc-5986-436f-8d2e-4550cbe5e49c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:52.330159 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.330022 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9555cbc-5986-436f-8d2e-4550cbe5e49c-kube-api-access-jx2fs\" (UniqueName: \"kubernetes.io/projected/d9555cbc-5986-436f-8d2e-4550cbe5e49c-kube-api-access-jx2fs\") pod \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " Oct 30 14:27:52.332048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.330280 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-model-cache\") pod \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\" (UID: \"d9555cbc-5986-436f-8d2e-4550cbe5e49c\") " Oct 30 14:27:52.332048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.330515 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-model-cache" pod "d9555cbc-5986-436f-8d2e-4550cbe5e49c" (UID: "d9555cbc-5986-436f-8d2e-4550cbe5e49c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:52.332048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.330694 4968 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.332048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.330706 4968 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.332048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.330740 4968 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.332048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.331228 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-dshm" pod "d9555cbc-5986-436f-8d2e-4550cbe5e49c" (UID: "d9555cbc-5986-436f-8d2e-4550cbe5e49c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:52.332443 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.332131 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tls-certs" pod "d9555cbc-5986-436f-8d2e-4550cbe5e49c" (UID: "d9555cbc-5986-436f-8d2e-4550cbe5e49c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:27:52.334323 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.334298 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9555cbc-5986-436f-8d2e-4550cbe5e49c-kube-api-access-jx2fs" pod "d9555cbc-5986-436f-8d2e-4550cbe5e49c" (UID: "d9555cbc-5986-436f-8d2e-4550cbe5e49c"). InnerVolumeSpecName "kube-api-access-jx2fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:52.402739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.392085 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 14:27:52.402739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.392147 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 14:27:52.432263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.432232 4968 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9555cbc-5986-436f-8d2e-4550cbe5e49c-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.432263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.432260 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx2fs\" (UniqueName: \"kubernetes.io/projected/d9555cbc-5986-436f-8d2e-4550cbe5e49c-kube-api-access-jx2fs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.432263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.432269 4968 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9555cbc-5986-436f-8d2e-4550cbe5e49c-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:27:52.765912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.765850 4968 generic.go:356] "Generic (PLEG): container finished" podID="d9555cbc-5986-436f-8d2e-4550cbe5e49c" containerID="47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4" exitCode=0 Oct 30 14:27:52.765912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.765899 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" Oct 30 14:27:52.766495 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.765933 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" event={"ID":"d9555cbc-5986-436f-8d2e-4550cbe5e49c","Type":"ContainerDied","Data":"47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4"} Oct 30 14:27:52.766495 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.765977 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq" event={"ID":"d9555cbc-5986-436f-8d2e-4550cbe5e49c","Type":"ContainerDied","Data":"7a5f23c906513261c05447d54a459a1541525f0ca93cc512a5ac8d244492c42a"} Oct 30 14:27:52.766495 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.766003 4968 scope.go:122] "RemoveContainer" containerID="47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4" Oct 30 14:27:52.767766 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.767738 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" event={"ID":"aca3c382-c905-4be5-ae1a-d5f9336adc1d","Type":"ContainerDied","Data":"2e1ede62e097bfc00ca12f8a08991751ac77863402c630baa9cf43f1f0a34d9f"} Oct 30 14:27:52.767914 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.767781 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks" Oct 30 14:27:52.777024 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.776994 4968 scope.go:122] "RemoveContainer" containerID="47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4" Oct 30 14:27:52.777285 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:27:52.777267 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4\": container with ID starting with 47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4 not found: ID does not exist" containerID="47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4" Oct 30 14:27:52.777343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.777294 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4"} err="failed to get container status \"47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4\": rpc error: code = NotFound desc = could not find container \"47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4\": container with ID starting with 47d77a60c1df76985979d7de71826f00ab86d980cfaa66a8fa782e029d02b7c4 not found: ID does not exist" Oct 30 14:27:52.777343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.777312 4968 scope.go:122] "RemoveContainer" containerID="4a4ece1581879a3f8fe49c45f1de7155206b1d1d099a5294aad8502d4eda042e" Oct 30 14:27:52.847667 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.847632 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq"] Oct 30 14:27:52.851722 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.851681 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/distinct-w5-7bf15f-kserve-7cfb889d7-zbznq"] Oct 30 14:27:52.906797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.906764 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks"] Oct 30 14:27:52.915326 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:52.915299 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-7bf15f/premium-w5-7bf15f-kserve-68576f66d-vl8ks"] Oct 30 14:27:53.035389 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:53.035353 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx"] Oct 30 14:27:53.041203 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:53.041178 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fd4567-p74vx"] Oct 30 14:27:53.393099 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:53.393061 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 14:27:53.393395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:53.393367 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 14:27:54.035873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:54.035839 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2"] Oct 30 14:27:54.044739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:54.044698 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0226f211-dbff2"] Oct 30 14:27:54.218526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:54.218490 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" podUID="8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:27:54.391738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:54.387119 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 14:27:54.391738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:54.387181 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 14:27:54.520486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:54.520451 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca3c382-c905-4be5-ae1a-d5f9336adc1d" path="/var/lib/kubelet/pods/aca3c382-c905-4be5-ae1a-d5f9336adc1d/volumes" Oct 30 14:27:54.520911 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:54.520897 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad09e501-9c7e-4554-b17d-569e84446ced" path="/var/lib/kubelet/pods/ad09e501-9c7e-4554-b17d-569e84446ced/volumes" Oct 30 14:27:54.521306 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:54.521293 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9555cbc-5986-436f-8d2e-4550cbe5e49c" path="/var/lib/kubelet/pods/d9555cbc-5986-436f-8d2e-4550cbe5e49c/volumes" Oct 30 14:27:54.521973 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:54.521958 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d975f97b-6681-4fe3-90a5-4d0b84590e70" path="/var/lib/kubelet/pods/d975f97b-6681-4fe3-90a5-4d0b84590e70/volumes" Oct 30 14:27:55.386305 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:55.386270 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 14:27:55.386712 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:55.386332 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 14:27:56.390297 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:56.390262 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:56.390831 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:56.390734 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:56.391430 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:56.391406 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:27:57.465082 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:57.463261 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4"] Oct 30 14:27:57.465082 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:57.463491 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" podUID="64ffd442-9b53-4b73-9af8-2fa54d148b78" containerName="istio-proxy" containerID="cri-o://0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f" gracePeriod=30 Oct 30 14:27:57.468835 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:57.468790 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn"] Oct 30 14:27:57.469160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:57.469135 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" containerID="cri-o://872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3" gracePeriod=30 Oct 30 14:27:57.473102 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:57.473081 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh"] Oct 30 14:27:57.473451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:57.473425 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" containerID="cri-o://c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38" gracePeriod=30 Oct 30 14:27:57.478310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:57.478288 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65"] Oct 30 14:27:57.478657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:57.478627 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" containerID="cri-o://79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33" gracePeriod=30 Oct 30 14:27:57.600838 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:57.599452 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn"] Oct 30 14:27:57.603848 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:57.603779 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7bf15f-5b5mn"] Oct 30 14:27:58.555570 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:58.555530 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894b4b7c-3a69-4649-a000-f45598e52d4d" path="/var/lib/kubelet/pods/894b4b7c-3a69-4649-a000-f45598e52d4d/volumes" Oct 30 14:27:58.847739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:58.840898 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:27:58.847739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:58.840982 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:27:59.045739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:59.045148 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 14:27:59.045739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:59.045204 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:27:59.059768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:59.058672 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd"] Oct 30 14:27:59.076418 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:27:59.076352 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f781596b-gwkqd"] Oct 30 14:28:00.522461 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:00.522418 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5a75e3-7932-4f81-bd14-c8b7d0e1166f" path="/var/lib/kubelet/pods/9e5a75e3-7932-4f81-bd14-c8b7d0e1166f/volumes" Oct 30 14:28:01.089538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:01.089506 4968 patch_prober.go:29] interesting pod/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:28:01.089707 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:01.089570 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:28:02.774847 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.774821 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:28:02.809459 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.809429 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:28:02.871264 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.871236 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:28:02.877528 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.873445 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:28:02.904680 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.904649 4968 generic.go:356] "Generic (PLEG): container finished" podID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerID="872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3" exitCode=0 Oct 30 14:28:02.904884 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.904690 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" Oct 30 14:28:02.904884 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.904743 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" event={"ID":"3dd4fc0f-9b38-4861-8869-03dc19722864","Type":"ContainerDied","Data":"872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3"} Oct 30 14:28:02.904884 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.904796 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn" event={"ID":"3dd4fc0f-9b38-4861-8869-03dc19722864","Type":"ContainerDied","Data":"feda7ebe93b9937837906766aaf76ed3a86eadcd6c90f89146a744dd8d014eef"} Oct 30 14:28:02.904884 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.904822 4968 scope.go:122] "RemoveContainer" containerID="872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3" Oct 30 14:28:02.906387 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.906364 4968 generic.go:356] "Generic (PLEG): container finished" podID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerID="c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38" exitCode=0 Oct 30 14:28:02.906509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.906404 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" Oct 30 14:28:02.906509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.906438 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" event={"ID":"e294e23e-573e-4f62-a2b8-27a3dd0009b7","Type":"ContainerDied","Data":"c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38"} Oct 30 14:28:02.906509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.906477 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh" event={"ID":"e294e23e-573e-4f62-a2b8-27a3dd0009b7","Type":"ContainerDied","Data":"812c5fb5522ec987ca09ab1b04f17afeab2aca93d3882c0080bd7553b63e421b"} Oct 30 14:28:02.908444 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.908421 4968 generic.go:356] "Generic (PLEG): container finished" podID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerID="79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33" exitCode=0 Oct 30 14:28:02.908583 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.908470 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" Oct 30 14:28:02.908583 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.908482 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" event={"ID":"41493a9c-2413-46f7-ae3d-1c79505299b4","Type":"ContainerDied","Data":"79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33"} Oct 30 14:28:02.908583 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.908523 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65" event={"ID":"41493a9c-2413-46f7-ae3d-1c79505299b4","Type":"ContainerDied","Data":"4de968a8497a43b99aa6c38eb60926adcaa2024644e76735af79dfa89798d828"} Oct 30 14:28:02.909539 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909520 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-data\") pod \"64ffd442-9b53-4b73-9af8-2fa54d148b78\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " Oct 30 14:28:02.909669 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909566 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-certs\") pod \"3dd4fc0f-9b38-4861-8869-03dc19722864\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " Oct 30 14:28:02.909669 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909584 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-socket\") pod \"3dd4fc0f-9b38-4861-8869-03dc19722864\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " Oct 30 14:28:02.909669 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909616 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-token\" (UniqueName: \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-token\") pod \"64ffd442-9b53-4b73-9af8-2fa54d148b78\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " Oct 30 14:28:02.909669 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909637 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-certs\") pod \"64ffd442-9b53-4b73-9af8-2fa54d148b78\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " Oct 30 14:28:02.909919 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909669 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-credential-socket\") pod \"3dd4fc0f-9b38-4861-8869-03dc19722864\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " Oct 30 14:28:02.909919 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909710 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ffd442-9b53-4b73-9af8-2fa54d148b78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ffd442-9b53-4b73-9af8-2fa54d148b78-istiod-ca-cert\") pod \"64ffd442-9b53-4b73-9af8-2fa54d148b78\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " Oct 30 14:28:02.909919 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909781 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-token\" (UniqueName: \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-token\") pod \"3dd4fc0f-9b38-4861-8869-03dc19722864\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " Oct 30 14:28:02.909919 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909815 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-podinfo\") pod \"64ffd442-9b53-4b73-9af8-2fa54d148b78\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " Oct 30 14:28:02.910405 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909818 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-data" pod "64ffd442-9b53-4b73-9af8-2fa54d148b78" (UID: "64ffd442-9b53-4b73-9af8-2fa54d148b78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:02.910497 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909920 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-certs" pod "3dd4fc0f-9b38-4861-8869-03dc19722864" (UID: "3dd4fc0f-9b38-4861-8869-03dc19722864"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:02.910497 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.909879 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-data\") pod \"3dd4fc0f-9b38-4861-8869-03dc19722864\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " Oct 30 14:28:02.910497 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910471 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-socket\") pod \"64ffd442-9b53-4b73-9af8-2fa54d148b78\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " Oct 30 14:28:02.910656 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910130 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-data" pod "3dd4fc0f-9b38-4861-8869-03dc19722864" (UID: "3dd4fc0f-9b38-4861-8869-03dc19722864"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:02.910656 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910203 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-credential-socket" pod "3dd4fc0f-9b38-4861-8869-03dc19722864" (UID: "3dd4fc0f-9b38-4861-8869-03dc19722864"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:02.910656 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910604 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-podinfo\") pod \"3dd4fc0f-9b38-4861-8869-03dc19722864\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " Oct 30 14:28:02.910656 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910623 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" event={"ID":"64ffd442-9b53-4b73-9af8-2fa54d148b78","Type":"ContainerDied","Data":"0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f"} Oct 30 14:28:02.910656 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910649 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-socket" pod "64ffd442-9b53-4b73-9af8-2fa54d148b78" (UID: "64ffd442-9b53-4b73-9af8-2fa54d148b78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:02.910928 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910705 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" Oct 30 14:28:02.910928 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910270 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ffd442-9b53-4b73-9af8-2fa54d148b78-istiod-ca-cert" pod "64ffd442-9b53-4b73-9af8-2fa54d148b78" (UID: "64ffd442-9b53-4b73-9af8-2fa54d148b78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:02.910928 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910693 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-kube-api-access-7t2zb\" (UniqueName: \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-kube-api-access-7t2zb\") pod \"64ffd442-9b53-4b73-9af8-2fa54d148b78\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " Oct 30 14:28:02.910928 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910903 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-kube-api-access-c58c8\" (UniqueName: \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-kube-api-access-c58c8\") pod \"3dd4fc0f-9b38-4861-8869-03dc19722864\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " Oct 30 14:28:02.911138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910970 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dd4fc0f-9b38-4861-8869-03dc19722864-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd4fc0f-9b38-4861-8869-03dc19722864-istiod-ca-cert\") pod \"3dd4fc0f-9b38-4861-8869-03dc19722864\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " Oct 30 14:28:02.911138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.911037 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-credential-socket\") pod \"64ffd442-9b53-4b73-9af8-2fa54d148b78\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " Oct 30 14:28:02.911138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.911076 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-envoy\") pod \"3dd4fc0f-9b38-4861-8869-03dc19722864\" (UID: \"3dd4fc0f-9b38-4861-8869-03dc19722864\") " Oct 30 14:28:02.911138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.911132 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-envoy\") pod \"64ffd442-9b53-4b73-9af8-2fa54d148b78\" (UID: \"64ffd442-9b53-4b73-9af8-2fa54d148b78\") " Oct 30 14:28:02.911395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910603 4968 generic.go:356] "Generic (PLEG): container finished" podID="64ffd442-9b53-4b73-9af8-2fa54d148b78" containerID="0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f" exitCode=0 Oct 30 14:28:02.911395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.911236 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4" event={"ID":"64ffd442-9b53-4b73-9af8-2fa54d148b78","Type":"ContainerDied","Data":"36f1101fe129c0faf866d21a4c2e4131e0e3d3b8624830eb64e4a6bca9cda711"} Oct 30 14:28:02.911395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910377 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-certs" pod "64ffd442-9b53-4b73-9af8-2fa54d148b78" (UID: "64ffd442-9b53-4b73-9af8-2fa54d148b78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:02.911395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.910422 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-socket" pod "3dd4fc0f-9b38-4861-8869-03dc19722864" (UID: "3dd4fc0f-9b38-4861-8869-03dc19722864"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:02.911642 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.911592 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-credential-socket" pod "64ffd442-9b53-4b73-9af8-2fa54d148b78" (UID: "64ffd442-9b53-4b73-9af8-2fa54d148b78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:02.911959 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.911935 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd4fc0f-9b38-4861-8869-03dc19722864-istiod-ca-cert" pod "3dd4fc0f-9b38-4861-8869-03dc19722864" (UID: "3dd4fc0f-9b38-4861-8869-03dc19722864"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:02.913085 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.913064 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-podinfo" pod "3dd4fc0f-9b38-4861-8869-03dc19722864" (UID: "3dd4fc0f-9b38-4861-8869-03dc19722864"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:28:02.914003 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.913985 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:02.914095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914008 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:02.914095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914023 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ffd442-9b53-4b73-9af8-2fa54d148b78-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:02.914095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914035 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:02.914095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914048 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:02.914095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914063 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:02.914095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914076 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd4fc0f-9b38-4861-8869-03dc19722864-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:02.914095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914088 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:02.914331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914102 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:02.914331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914116 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:02.914331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914131 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:02.914331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914186 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-envoy" pod "3dd4fc0f-9b38-4861-8869-03dc19722864" (UID: "3dd4fc0f-9b38-4861-8869-03dc19722864"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:02.914648 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.914631 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-envoy" pod "64ffd442-9b53-4b73-9af8-2fa54d148b78" (UID: "64ffd442-9b53-4b73-9af8-2fa54d148b78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:02.915060 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.915036 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-token" pod "64ffd442-9b53-4b73-9af8-2fa54d148b78" (UID: "64ffd442-9b53-4b73-9af8-2fa54d148b78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:02.915166 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.915066 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-podinfo" pod "64ffd442-9b53-4b73-9af8-2fa54d148b78" (UID: "64ffd442-9b53-4b73-9af8-2fa54d148b78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:28:02.915166 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.915072 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-kube-api-access-7t2zb" pod "64ffd442-9b53-4b73-9af8-2fa54d148b78" (UID: "64ffd442-9b53-4b73-9af8-2fa54d148b78"). InnerVolumeSpecName "kube-api-access-7t2zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:02.915166 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.915063 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-token" pod "3dd4fc0f-9b38-4861-8869-03dc19722864" (UID: "3dd4fc0f-9b38-4861-8869-03dc19722864"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:02.916279 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.916255 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-kube-api-access-c58c8" pod "3dd4fc0f-9b38-4861-8869-03dc19722864" (UID: "3dd4fc0f-9b38-4861-8869-03dc19722864"). InnerVolumeSpecName "kube-api-access-c58c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:02.917753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.917706 4968 scope.go:122] "RemoveContainer" containerID="872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3" Oct 30 14:28:02.918022 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:02.918000 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3\": container with ID starting with 872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3 not found: ID does not exist" containerID="872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3" Oct 30 14:28:02.918094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.918029 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3"} err="failed to get container status \"872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3\": rpc error: code = NotFound desc = could not find container \"872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3\": container with ID starting with 872c1e8639a6efdd0675ecee0d1400e19e0a8875ffc5f2f160dce9b4f8a617a3 not found: ID does not exist" Oct 30 14:28:02.918094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.918052 4968 scope.go:122] "RemoveContainer" containerID="c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38" Oct 30 14:28:02.929467 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.929433 4968 scope.go:122] "RemoveContainer" containerID="c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38" Oct 30 14:28:02.929797 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:02.929767 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38\": container with ID starting with c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38 not found: ID does not exist" containerID="c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38" Oct 30 14:28:02.929901 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.929808 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38"} err="failed to get container status \"c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38\": rpc error: code = NotFound desc = could not find container \"c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38\": container with ID starting with c6c5bf2be6126f5470cfb7110f7f53adf8efb68f3bf04cc2ae6e764944042e38 not found: ID does not exist" Oct 30 14:28:02.929901 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.929832 4968 scope.go:122] "RemoveContainer" containerID="79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33" Oct 30 14:28:02.942318 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.942284 4968 scope.go:122] "RemoveContainer" containerID="cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e" Oct 30 14:28:02.952260 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.952236 4968 scope.go:122] "RemoveContainer" containerID="79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33" Oct 30 14:28:02.952564 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:02.952541 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33\": container with ID starting with 79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33 not found: ID does not exist" containerID="79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33" Oct 30 14:28:02.952660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.952572 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33"} err="failed to get container status \"79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33\": rpc error: code = NotFound desc = could not find container \"79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33\": container with ID starting with 79aa2eab3f3fb5beaf702458a0dbefae9c9ee6d9f4562c37583b43f6146a1a33 not found: ID does not exist" Oct 30 14:28:02.952660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.952588 4968 scope.go:122] "RemoveContainer" containerID="cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e" Oct 30 14:28:02.952856 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:02.952838 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e\": container with ID starting with cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e not found: ID does not exist" containerID="cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e" Oct 30 14:28:02.952923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.952861 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e"} err="failed to get container status \"cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e\": rpc error: code = NotFound desc = could not find container \"cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e\": container with ID starting with cb4d11b5364c701e89515c04c4f8b34de8b27790b1c11b6dd1c2f7f0f8db7c4e not found: ID does not exist" Oct 30 14:28:02.952923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.952874 4968 scope.go:122] "RemoveContainer" containerID="0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f" Oct 30 14:28:02.964217 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.964195 4968 scope.go:122] "RemoveContainer" containerID="0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f" Oct 30 14:28:02.964506 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:02.964487 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f\": container with ID starting with 0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f not found: ID does not exist" containerID="0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f" Oct 30 14:28:02.964585 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:02.964512 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f"} err="failed to get container status \"0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f\": rpc error: code = NotFound desc = could not find container \"0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f\": container with ID starting with 0810d2e60c6542eec4fab69e880b91dabdbad02439150c9b297d489d5063274f not found: ID does not exist" Oct 30 14:28:03.015059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.014978 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-data\") pod \"41493a9c-2413-46f7-ae3d-1c79505299b4\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " Oct 30 14:28:03.015059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015014 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-credential-socket\") pod \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " Oct 30 14:28:03.015059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015042 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-podinfo\") pod \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " Oct 30 14:28:03.015346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015079 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-socket\") pod \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " Oct 30 14:28:03.015346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015098 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-envoy\") pod \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " Oct 30 14:28:03.015346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015137 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-kube-api-access-vrtff\" (UniqueName: \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-kube-api-access-vrtff\") pod \"41493a9c-2413-46f7-ae3d-1c79505299b4\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " Oct 30 14:28:03.015346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015231 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-socket\") pod \"41493a9c-2413-46f7-ae3d-1c79505299b4\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " Oct 30 14:28:03.015346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015273 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-envoy\") pod \"41493a9c-2413-46f7-ae3d-1c79505299b4\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " Oct 30 14:28:03.015346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015300 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-certs\") pod \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " Oct 30 14:28:03.015346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015296 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-credential-socket" pod "e294e23e-573e-4f62-a2b8-27a3dd0009b7" (UID: "e294e23e-573e-4f62-a2b8-27a3dd0009b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:03.015346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015342 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-kube-api-access-7l8jl\" (UniqueName: \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-kube-api-access-7l8jl\") pod \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " Oct 30 14:28:03.015805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015300 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-data" pod "41493a9c-2413-46f7-ae3d-1c79505299b4" (UID: "41493a9c-2413-46f7-ae3d-1c79505299b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:03.015805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015374 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istiod-ca-cert\") pod \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " Oct 30 14:28:03.015805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015389 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-socket" pod "e294e23e-573e-4f62-a2b8-27a3dd0009b7" (UID: "e294e23e-573e-4f62-a2b8-27a3dd0009b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:03.015805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015413 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-token\" (UniqueName: \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-token\") pod \"41493a9c-2413-46f7-ae3d-1c79505299b4\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " Oct 30 14:28:03.015805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015447 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-credential-socket\") pod \"41493a9c-2413-46f7-ae3d-1c79505299b4\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " Oct 30 14:28:03.015805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015515 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-podinfo\") pod \"41493a9c-2413-46f7-ae3d-1c79505299b4\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " Oct 30 14:28:03.015805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015550 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-certs\") pod \"41493a9c-2413-46f7-ae3d-1c79505299b4\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " Oct 30 14:28:03.015805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015589 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41493a9c-2413-46f7-ae3d-1c79505299b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41493a9c-2413-46f7-ae3d-1c79505299b4-istiod-ca-cert\") pod \"41493a9c-2413-46f7-ae3d-1c79505299b4\" (UID: \"41493a9c-2413-46f7-ae3d-1c79505299b4\") " Oct 30 14:28:03.015805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015629 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-data\") pod \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " Oct 30 14:28:03.015805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015679 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-token\" (UniqueName: \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-token\") pod \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\" (UID: \"e294e23e-573e-4f62-a2b8-27a3dd0009b7\") " Oct 30 14:28:03.016354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016205 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.016354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016230 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.016354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016245 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.016354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016260 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.016354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016273 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.016354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016287 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.016354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016302 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.016354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016315 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ffd442-9b53-4b73-9af8-2fa54d148b78-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.016354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016330 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t2zb\" (UniqueName: \"kubernetes.io/projected/64ffd442-9b53-4b73-9af8-2fa54d148b78-kube-api-access-7t2zb\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.016354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016343 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c58c8\" (UniqueName: \"kubernetes.io/projected/3dd4fc0f-9b38-4861-8869-03dc19722864-kube-api-access-c58c8\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.016897 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015547 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-socket" pod "41493a9c-2413-46f7-ae3d-1c79505299b4" (UID: "41493a9c-2413-46f7-ae3d-1c79505299b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:03.016897 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015704 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-certs" pod "e294e23e-573e-4f62-a2b8-27a3dd0009b7" (UID: "e294e23e-573e-4f62-a2b8-27a3dd0009b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:03.016897 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.015975 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-certs" pod "41493a9c-2413-46f7-ae3d-1c79505299b4" (UID: "41493a9c-2413-46f7-ae3d-1c79505299b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:03.016897 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016161 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-credential-socket" pod "41493a9c-2413-46f7-ae3d-1c79505299b4" (UID: "41493a9c-2413-46f7-ae3d-1c79505299b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:03.016897 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016839 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41493a9c-2413-46f7-ae3d-1c79505299b4-istiod-ca-cert" pod "41493a9c-2413-46f7-ae3d-1c79505299b4" (UID: "41493a9c-2413-46f7-ae3d-1c79505299b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:03.017160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.016955 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-data" pod "e294e23e-573e-4f62-a2b8-27a3dd0009b7" (UID: "e294e23e-573e-4f62-a2b8-27a3dd0009b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:03.017160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.017107 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istiod-ca-cert" pod "e294e23e-573e-4f62-a2b8-27a3dd0009b7" (UID: "e294e23e-573e-4f62-a2b8-27a3dd0009b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:03.018300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.018027 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-podinfo" pod "e294e23e-573e-4f62-a2b8-27a3dd0009b7" (UID: "e294e23e-573e-4f62-a2b8-27a3dd0009b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:28:03.018300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.018064 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-envoy" pod "e294e23e-573e-4f62-a2b8-27a3dd0009b7" (UID: "e294e23e-573e-4f62-a2b8-27a3dd0009b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:03.018300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.018124 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-token" pod "41493a9c-2413-46f7-ae3d-1c79505299b4" (UID: "41493a9c-2413-46f7-ae3d-1c79505299b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:03.018300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.018203 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-kube-api-access-7l8jl" pod "e294e23e-573e-4f62-a2b8-27a3dd0009b7" (UID: "e294e23e-573e-4f62-a2b8-27a3dd0009b7"). InnerVolumeSpecName "kube-api-access-7l8jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:03.018670 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.018650 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-token" pod "e294e23e-573e-4f62-a2b8-27a3dd0009b7" (UID: "e294e23e-573e-4f62-a2b8-27a3dd0009b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:03.019313 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.019291 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-envoy" pod "41493a9c-2413-46f7-ae3d-1c79505299b4" (UID: "41493a9c-2413-46f7-ae3d-1c79505299b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:03.019431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.019323 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-podinfo" pod "41493a9c-2413-46f7-ae3d-1c79505299b4" (UID: "41493a9c-2413-46f7-ae3d-1c79505299b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:28:03.019871 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.019851 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-kube-api-access-vrtff" pod "41493a9c-2413-46f7-ae3d-1c79505299b4" (UID: "41493a9c-2413-46f7-ae3d-1c79505299b4"). InnerVolumeSpecName "kube-api-access-vrtff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:03.117989 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.117956 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.117989 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.117988 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118003 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrtff\" (UniqueName: \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-kube-api-access-vrtff\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118017 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118032 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118047 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118059 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l8jl\" (UniqueName: \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-kube-api-access-7l8jl\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118072 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118088 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118102 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118118 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41493a9c-2413-46f7-ae3d-1c79505299b4-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118131 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41493a9c-2413-46f7-ae3d-1c79505299b4-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118143 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41493a9c-2413-46f7-ae3d-1c79505299b4-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118155 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.118163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.118166 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e294e23e-573e-4f62-a2b8-27a3dd0009b7-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:03.260682 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.260650 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh"] Oct 30 14:28:03.265293 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.265224 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-t6kjh"] Oct 30 14:28:03.297203 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.297175 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn"] Oct 30 14:28:03.303201 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.303176 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-44prn"] Oct 30 14:28:03.336106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.336074 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65"] Oct 30 14:28:03.344676 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.344645 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-9bl65"] Oct 30 14:28:03.374731 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.374676 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4"] Oct 30 14:28:03.381361 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:03.381332 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-137f01-openshift-default-59bc5fc5d9-m67d4"] Oct 30 14:28:04.217914 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:04.217875 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" podUID="8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:28:04.340759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:04.340710 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:28:04.542789 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:04.532734 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" path="/var/lib/kubelet/pods/3dd4fc0f-9b38-4861-8869-03dc19722864/volumes" Oct 30 14:28:04.542789 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:04.533388 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" path="/var/lib/kubelet/pods/41493a9c-2413-46f7-ae3d-1c79505299b4/volumes" Oct 30 14:28:04.548590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:04.548556 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ffd442-9b53-4b73-9af8-2fa54d148b78" path="/var/lib/kubelet/pods/64ffd442-9b53-4b73-9af8-2fa54d148b78/volumes" Oct 30 14:28:04.549316 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:04.549223 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" path="/var/lib/kubelet/pods/e294e23e-573e-4f62-a2b8-27a3dd0009b7/volumes" Oct 30 14:28:05.308911 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:05.308860 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc"] Oct 30 14:28:05.309384 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:05.309098 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerName="istio-proxy" containerID="cri-o://77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46" gracePeriod=30 Oct 30 14:28:05.322639 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:05.322610 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7"] Oct 30 14:28:05.322899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:05.322873 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" containerID="cri-o://9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a" gracePeriod=30 Oct 30 14:28:05.326161 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:05.326135 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7"] Oct 30 14:28:05.326322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:05.326304 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" containerID="cri-o://0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be" gracePeriod=30 Oct 30 14:28:05.337749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:05.337710 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl"] Oct 30 14:28:05.337899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:05.337882 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerName="istio-proxy" containerID="cri-o://50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606" gracePeriod=30 Oct 30 14:28:05.942751 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:05.942515 4968 generic.go:356] "Generic (PLEG): container finished" podID="8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6" containerID="c230a8f34fc1f286942a0e74048b0cd1341d3ec9c5e45972d919ca0b1efd7934" exitCode=0 Oct 30 14:28:05.942751 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:05.942580 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" event={"ID":"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6","Type":"ContainerDied","Data":"c230a8f34fc1f286942a0e74048b0cd1341d3ec9c5e45972d919ca0b1efd7934"} Oct 30 14:28:06.036496 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.036464 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv"] Oct 30 14:28:06.045382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.045352 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493967ac-rkdnv"] Oct 30 14:28:06.064252 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.064228 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:28:06.162805 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.162775 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-plugins-config-volume\") pod \"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6\" (UID: \"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6\") " Oct 30 14:28:06.162992 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.162883 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-kube-api-access-lg6xf\" (UniqueName: \"kubernetes.io/projected/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-kube-api-access-lg6xf\") pod \"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6\" (UID: \"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6\") " Oct 30 14:28:06.163188 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.163156 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-plugins-config-volume" pod "8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6" (UID: "8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:06.165147 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.165125 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-kube-api-access-lg6xf" pod "8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6" (UID: "8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6"). InnerVolumeSpecName "kube-api-access-lg6xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:06.264042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.264005 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg6xf\" (UniqueName: \"kubernetes.io/projected/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-kube-api-access-lg6xf\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:06.264042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.264040 4968 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:06.518562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.518485 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1a5ba3-86a2-4784-873e-e33d00fc5529" path="/var/lib/kubelet/pods/9f1a5ba3-86a2-4784-873e-e33d00fc5529/volumes" Oct 30 14:28:06.952316 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.952285 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" Oct 30 14:28:06.952520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.952321 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4" event={"ID":"8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6","Type":"ContainerDied","Data":"26d31ef935c2d9a37ab586e2b4819d3f84615ff2e164aaf63f074b917dc6291f"} Oct 30 14:28:06.952520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:06.952396 4968 scope.go:122] "RemoveContainer" containerID="c230a8f34fc1f286942a0e74048b0cd1341d3ec9c5e45972d919ca0b1efd7934" Oct 30 14:28:07.005339 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:07.005312 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4"] Oct 30 14:28:07.015192 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:07.015169 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7bf15f-b47b57d47-r59g4"] Oct 30 14:28:08.520070 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:08.520034 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6" path="/var/lib/kubelet/pods/8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6/volumes" Oct 30 14:28:08.929538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:08.929502 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:28:08.929757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:08.929573 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:28:10.201730 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.201679 4968 patch_prober.go:29] interesting pod/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:28:10.202122 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.201776 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:28:10.398373 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:10.398338 4968 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0614bcf9_f822_4d09_acd4_4ae4b55a2461.slice/crio-conmon-77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:28:10.603750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.602415 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:28:10.609275 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.609244 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-podinfo\") pod \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " Oct 30 14:28:10.609404 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.609296 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-token\" (UniqueName: \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-token\") pod \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " Oct 30 14:28:10.609404 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.609375 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-credential-socket\") pod \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " Oct 30 14:28:10.609504 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.609409 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-kube-api-access-xvgsk\" (UniqueName: \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-kube-api-access-xvgsk\") pod \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " Oct 30 14:28:10.609504 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.609455 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-socket\") pod \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " Oct 30 14:28:10.609609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.609500 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istiod-ca-cert\") pod \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " Oct 30 14:28:10.609609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.609531 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-data\") pod \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " Oct 30 14:28:10.609609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.609566 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-certs\") pod \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " Oct 30 14:28:10.612250 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.612211 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-certs" pod "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" (UID: "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.612415 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.612395 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-socket" pod "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" (UID: "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.612696 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.612676 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istiod-ca-cert" pod "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" (UID: "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:10.613503 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.613478 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-data" pod "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" (UID: "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.613675 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.613657 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-credential-socket" pod "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" (UID: "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.614448 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.614415 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-kube-api-access-xvgsk" pod "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" (UID: "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03"). InnerVolumeSpecName "kube-api-access-xvgsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:10.616907 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.616883 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-token" pod "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" (UID: "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:10.617060 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.617018 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-podinfo" pod "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" (UID: "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:28:10.668638 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.668609 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:28:10.685378 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.685354 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:28:10.701004 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.700979 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:28:10.712466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712379 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istiod-ca-cert\") pod \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " Oct 30 14:28:10.712466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712427 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-certs\") pod \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " Oct 30 14:28:10.712657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712467 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-certs\") pod \"047af57e-0edc-42c0-bb08-e5306dddce4a\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " Oct 30 14:28:10.712657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712517 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-data\") pod \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " Oct 30 14:28:10.712657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712552 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047af57e-0edc-42c0-bb08-e5306dddce4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047af57e-0edc-42c0-bb08-e5306dddce4a-istiod-ca-cert\") pod \"047af57e-0edc-42c0-bb08-e5306dddce4a\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " Oct 30 14:28:10.712657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712580 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-socket\") pod \"047af57e-0edc-42c0-bb08-e5306dddce4a\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " Oct 30 14:28:10.712657 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712647 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-podinfo\") pod \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " Oct 30 14:28:10.712952 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712683 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-podinfo\") pod \"047af57e-0edc-42c0-bb08-e5306dddce4a\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " Oct 30 14:28:10.712952 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712727 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-credential-socket\") pod \"047af57e-0edc-42c0-bb08-e5306dddce4a\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " Oct 30 14:28:10.712952 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712756 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-podinfo\") pod \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " Oct 30 14:28:10.712952 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712812 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-data\") pod \"047af57e-0edc-42c0-bb08-e5306dddce4a\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " Oct 30 14:28:10.712952 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712859 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-kube-api-access-bdksj\" (UniqueName: \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-kube-api-access-bdksj\") pod \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " Oct 30 14:28:10.712952 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712896 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-socket\") pod \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " Oct 30 14:28:10.712952 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712934 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-envoy\") pod \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " Oct 30 14:28:10.713310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712966 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-envoy\") pod \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\" (UID: \"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03\") " Oct 30 14:28:10.713310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.712997 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-kube-api-access-nnxbc\" (UniqueName: \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-kube-api-access-nnxbc\") pod \"047af57e-0edc-42c0-bb08-e5306dddce4a\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " Oct 30 14:28:10.713310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713028 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istiod-ca-cert\") pod \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " Oct 30 14:28:10.713310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713059 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-credential-socket\") pod \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " Oct 30 14:28:10.713310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713128 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-certs\") pod \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " Oct 30 14:28:10.713310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713156 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-credential-socket\") pod \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " Oct 30 14:28:10.713310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713192 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-envoy\") pod \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " Oct 30 14:28:10.713310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713219 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-token\" (UniqueName: \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-token\") pod \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " Oct 30 14:28:10.713310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713249 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-data\") pod \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " Oct 30 14:28:10.713310 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713290 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-token\" (UniqueName: \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-token\") pod \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\" (UID: \"0614bcf9-f822-4d09-acd4-4ae4b55a2461\") " Oct 30 14:28:10.713843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713347 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-socket\") pod \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " Oct 30 14:28:10.713843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713376 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-token\" (UniqueName: \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-token\") pod \"047af57e-0edc-42c0-bb08-e5306dddce4a\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " Oct 30 14:28:10.713843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713394 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-socket" pod "047af57e-0edc-42c0-bb08-e5306dddce4a" (UID: "047af57e-0edc-42c0-bb08-e5306dddce4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.713843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713397 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istiod-ca-cert" pod "0614bcf9-f822-4d09-acd4-4ae4b55a2461" (UID: "0614bcf9-f822-4d09-acd4-4ae4b55a2461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:10.713843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713404 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-envoy\") pod \"047af57e-0edc-42c0-bb08-e5306dddce4a\" (UID: \"047af57e-0edc-42c0-bb08-e5306dddce4a\") " Oct 30 14:28:10.713843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713480 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-kube-api-access-fbpbp\" (UniqueName: \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-kube-api-access-fbpbp\") pod \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\" (UID: \"a6abc3e1-e0d0-41f4-b180-f82afe240a07\") " Oct 30 14:28:10.713843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.713647 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-certs" pod "a6abc3e1-e0d0-41f4-b180-f82afe240a07" (UID: "a6abc3e1-e0d0-41f4-b180-f82afe240a07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.714590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.714263 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.714590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.714288 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.714590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.714303 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.714590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.714317 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.714590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.714332 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.714590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.714345 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvgsk\" (UniqueName: \"kubernetes.io/projected/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-kube-api-access-xvgsk\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.714590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.714358 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.714590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.714371 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.714590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.714386 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.714590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.714398 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.714590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.714411 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.716690 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.715425 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-data" pod "a6abc3e1-e0d0-41f4-b180-f82afe240a07" (UID: "a6abc3e1-e0d0-41f4-b180-f82afe240a07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.716690 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.715751 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047af57e-0edc-42c0-bb08-e5306dddce4a-istiod-ca-cert" pod "047af57e-0edc-42c0-bb08-e5306dddce4a" (UID: "047af57e-0edc-42c0-bb08-e5306dddce4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:10.716690 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.716053 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-kube-api-access-fbpbp" pod "a6abc3e1-e0d0-41f4-b180-f82afe240a07" (UID: "a6abc3e1-e0d0-41f4-b180-f82afe240a07"). InnerVolumeSpecName "kube-api-access-fbpbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:10.716690 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.716443 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-data" pod "047af57e-0edc-42c0-bb08-e5306dddce4a" (UID: "047af57e-0edc-42c0-bb08-e5306dddce4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.717635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.717020 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-certs" pod "047af57e-0edc-42c0-bb08-e5306dddce4a" (UID: "047af57e-0edc-42c0-bb08-e5306dddce4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.717744 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.717644 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-credential-socket" pod "047af57e-0edc-42c0-bb08-e5306dddce4a" (UID: "047af57e-0edc-42c0-bb08-e5306dddce4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.718528 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.718363 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-envoy" pod "047af57e-0edc-42c0-bb08-e5306dddce4a" (UID: "047af57e-0edc-42c0-bb08-e5306dddce4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.718875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.718663 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-credential-socket" pod "0614bcf9-f822-4d09-acd4-4ae4b55a2461" (UID: "0614bcf9-f822-4d09-acd4-4ae4b55a2461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.718875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.718792 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-envoy" pod "0614bcf9-f822-4d09-acd4-4ae4b55a2461" (UID: "0614bcf9-f822-4d09-acd4-4ae4b55a2461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.719020 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.718894 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-certs" pod "0614bcf9-f822-4d09-acd4-4ae4b55a2461" (UID: "0614bcf9-f822-4d09-acd4-4ae4b55a2461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.719087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.719036 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-socket" pod "0614bcf9-f822-4d09-acd4-4ae4b55a2461" (UID: "0614bcf9-f822-4d09-acd4-4ae4b55a2461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.719144 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.719101 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-credential-socket" pod "a6abc3e1-e0d0-41f4-b180-f82afe240a07" (UID: "a6abc3e1-e0d0-41f4-b180-f82afe240a07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.719571 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.719203 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-data" pod "0614bcf9-f822-4d09-acd4-4ae4b55a2461" (UID: "0614bcf9-f822-4d09-acd4-4ae4b55a2461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.719571 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.719335 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istiod-ca-cert" pod "a6abc3e1-e0d0-41f4-b180-f82afe240a07" (UID: "a6abc3e1-e0d0-41f4-b180-f82afe240a07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:10.719897 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.719734 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-socket" pod "a6abc3e1-e0d0-41f4-b180-f82afe240a07" (UID: "a6abc3e1-e0d0-41f4-b180-f82afe240a07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.720271 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.720180 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-token" pod "a6abc3e1-e0d0-41f4-b180-f82afe240a07" (UID: "a6abc3e1-e0d0-41f4-b180-f82afe240a07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:10.723045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.723024 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-podinfo" pod "0614bcf9-f822-4d09-acd4-4ae4b55a2461" (UID: "0614bcf9-f822-4d09-acd4-4ae4b55a2461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:28:10.723466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.723427 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-envoy" pod "a6abc3e1-e0d0-41f4-b180-f82afe240a07" (UID: "a6abc3e1-e0d0-41f4-b180-f82afe240a07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.723466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.723450 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-token" pod "047af57e-0edc-42c0-bb08-e5306dddce4a" (UID: "047af57e-0edc-42c0-bb08-e5306dddce4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:10.723624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.723590 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-podinfo" pod "a6abc3e1-e0d0-41f4-b180-f82afe240a07" (UID: "a6abc3e1-e0d0-41f4-b180-f82afe240a07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:28:10.723706 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.723674 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-token" pod "0614bcf9-f822-4d09-acd4-4ae4b55a2461" (UID: "0614bcf9-f822-4d09-acd4-4ae4b55a2461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:10.723802 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.723763 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-kube-api-access-bdksj" pod "0614bcf9-f822-4d09-acd4-4ae4b55a2461" (UID: "0614bcf9-f822-4d09-acd4-4ae4b55a2461"). InnerVolumeSpecName "kube-api-access-bdksj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:10.724120 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.724097 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-podinfo" pod "047af57e-0edc-42c0-bb08-e5306dddce4a" (UID: "047af57e-0edc-42c0-bb08-e5306dddce4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:28:10.724838 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.724815 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-kube-api-access-nnxbc" pod "047af57e-0edc-42c0-bb08-e5306dddce4a" (UID: "047af57e-0edc-42c0-bb08-e5306dddce4a"). InnerVolumeSpecName "kube-api-access-nnxbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:10.725075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.725058 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-envoy" pod "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" (UID: "1c0ef2f1-6d69-45c0-bfe1-60438eeefb03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:10.815786 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815749 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815786 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815787 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815803 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815846 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815861 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815874 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815886 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815898 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815912 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815924 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815938 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbpbp\" (UniqueName: \"kubernetes.io/projected/a6abc3e1-e0d0-41f4-b180-f82afe240a07-kube-api-access-fbpbp\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815952 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815967 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815979 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047af57e-0edc-42c0-bb08-e5306dddce4a-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.815988 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.815992 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.816398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.816004 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.816398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.816020 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.816398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.816033 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0614bcf9-f822-4d09-acd4-4ae4b55a2461-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.816398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.816046 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047af57e-0edc-42c0-bb08-e5306dddce4a-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.816398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.816060 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdksj\" (UniqueName: \"kubernetes.io/projected/0614bcf9-f822-4d09-acd4-4ae4b55a2461-kube-api-access-bdksj\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.816398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.816073 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0614bcf9-f822-4d09-acd4-4ae4b55a2461-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.816398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.816086 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.816398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.816099 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.816398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.816111 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnxbc\" (UniqueName: \"kubernetes.io/projected/047af57e-0edc-42c0-bb08-e5306dddce4a-kube-api-access-nnxbc\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.816398 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.816125 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6abc3e1-e0d0-41f4-b180-f82afe240a07-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:10.986287 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.986198 4968 generic.go:356] "Generic (PLEG): container finished" podID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerID="77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46" exitCode=0 Oct 30 14:28:10.986287 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.986250 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" Oct 30 14:28:10.986497 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.986291 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" event={"ID":"0614bcf9-f822-4d09-acd4-4ae4b55a2461","Type":"ContainerDied","Data":"77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46"} Oct 30 14:28:10.986497 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.986355 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc" event={"ID":"0614bcf9-f822-4d09-acd4-4ae4b55a2461","Type":"ContainerDied","Data":"9c62f3ca7ab55f27b1aa7696d61c098924c74bf5034751d268fc533c5961346d"} Oct 30 14:28:10.986497 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.986387 4968 scope.go:122] "RemoveContainer" containerID="77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46" Oct 30 14:28:10.988019 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.987992 4968 generic.go:356] "Generic (PLEG): container finished" podID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerID="9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a" exitCode=0 Oct 30 14:28:10.988169 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.988111 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" event={"ID":"a6abc3e1-e0d0-41f4-b180-f82afe240a07","Type":"ContainerDied","Data":"9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a"} Oct 30 14:28:10.988169 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.988152 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" event={"ID":"a6abc3e1-e0d0-41f4-b180-f82afe240a07","Type":"ContainerDied","Data":"915e98579684f550d2f6ab3f7719412bae38b3c3f3abd75f97020936243df135"} Oct 30 14:28:10.988290 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.988219 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7" Oct 30 14:28:10.996384 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.996359 4968 generic.go:356] "Generic (PLEG): container finished" podID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerID="50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606" exitCode=0 Oct 30 14:28:10.996491 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.996402 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" Oct 30 14:28:10.996563 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.996495 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" event={"ID":"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03","Type":"ContainerDied","Data":"50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606"} Oct 30 14:28:10.996563 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.996533 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl" event={"ID":"1c0ef2f1-6d69-45c0-bfe1-60438eeefb03","Type":"ContainerDied","Data":"8070089dc2bbf2cc2d0210b69439180e511e5b73ba721ca66fc756269b7114a4"} Oct 30 14:28:10.997963 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.997880 4968 generic.go:356] "Generic (PLEG): container finished" podID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerID="0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be" exitCode=0 Oct 30 14:28:10.997963 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.997937 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" Oct 30 14:28:10.998125 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.998021 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" event={"ID":"047af57e-0edc-42c0-bb08-e5306dddce4a","Type":"ContainerDied","Data":"0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be"} Oct 30 14:28:10.998125 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.998064 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7" event={"ID":"047af57e-0edc-42c0-bb08-e5306dddce4a","Type":"ContainerDied","Data":"969e7670383e3d1c47a59d89ee36e281e6fe44b00852c55ac82e5d3430ce45bd"} Oct 30 14:28:10.998942 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.998920 4968 scope.go:122] "RemoveContainer" containerID="77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46" Oct 30 14:28:10.999253 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:10.999232 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46\": container with ID starting with 77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46 not found: ID does not exist" containerID="77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46" Oct 30 14:28:10.999335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.999262 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46"} err="failed to get container status \"77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46\": rpc error: code = NotFound desc = could not find container \"77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46\": container with ID starting with 77ca758dfa189cff042bb22d28a6aedc64a4b75b2ea52497edb626fbf2352c46 not found: ID does not exist" Oct 30 14:28:10.999335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:10.999284 4968 scope.go:122] "RemoveContainer" containerID="9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a" Oct 30 14:28:11.014466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.014445 4968 scope.go:122] "RemoveContainer" containerID="9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a" Oct 30 14:28:11.015242 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:11.015220 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a\": container with ID starting with 9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a not found: ID does not exist" containerID="9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a" Oct 30 14:28:11.015347 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.015246 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a"} err="failed to get container status \"9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a\": rpc error: code = NotFound desc = could not find container \"9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a\": container with ID starting with 9a786e99f1106283f9fb493498f31d4edc68e309f71f91c35532fca4325d859a not found: ID does not exist" Oct 30 14:28:11.015347 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.015263 4968 scope.go:122] "RemoveContainer" containerID="50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606" Oct 30 14:28:11.027464 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.027434 4968 scope.go:122] "RemoveContainer" containerID="50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606" Oct 30 14:28:11.027779 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:11.027760 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606\": container with ID starting with 50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606 not found: ID does not exist" containerID="50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606" Oct 30 14:28:11.027866 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.027785 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606"} err="failed to get container status \"50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606\": rpc error: code = NotFound desc = could not find container \"50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606\": container with ID starting with 50a3d746e57e12e9c7652dea9e1950e2046de79511e34d4beb4c6f97efe3c606 not found: ID does not exist" Oct 30 14:28:11.027866 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.027801 4968 scope.go:122] "RemoveContainer" containerID="0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be" Oct 30 14:28:11.037147 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.037129 4968 scope.go:122] "RemoveContainer" containerID="0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be" Oct 30 14:28:11.037397 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:11.037376 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be\": container with ID starting with 0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be not found: ID does not exist" containerID="0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be" Oct 30 14:28:11.037468 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.037404 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be"} err="failed to get container status \"0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be\": rpc error: code = NotFound desc = could not find container \"0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be\": container with ID starting with 0a865e6adf553c97f8db1917c5f0ef630b5d08c224240fca049ca9cb05f705be not found: ID does not exist" Oct 30 14:28:11.065228 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.065203 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc"] Oct 30 14:28:11.069650 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.069626 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-f5fmc"] Oct 30 14:28:11.114042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.113983 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7"] Oct 30 14:28:11.117253 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.117231 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-mbsh7"] Oct 30 14:28:11.146439 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.146415 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl"] Oct 30 14:28:11.150642 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.150619 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-ls9nl"] Oct 30 14:28:11.183974 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.183943 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7"] Oct 30 14:28:11.188509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.188484 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7bf15f-openshift-default-7b7965884f-rmcx7"] Oct 30 14:28:11.687023 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.686943 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67c6566688-5rdmr"] Oct 30 14:28:11.687376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687346 4968 cpu_manager.go:414] "containerMap: removing container" podUID="35fa2371-a482-470e-b019-baa5cd14b25c" containerName="revoke-keys" Oct 30 14:28:11.687376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687357 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fa2371-a482-470e-b019-baa5cd14b25c" containerName="revoke-keys" Oct 30 14:28:11.687376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687372 4968 cpu_manager.go:414] "containerMap: removing container" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687377 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687384 4968 cpu_manager.go:414] "containerMap: removing container" podUID="d9555cbc-5986-436f-8d2e-4550cbe5e49c" containerName="main" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687389 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9555cbc-5986-436f-8d2e-4550cbe5e49c" containerName="main" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687398 4968 cpu_manager.go:414] "containerMap: removing container" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687402 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687411 4968 cpu_manager.go:414] "containerMap: removing container" podUID="64ffd442-9b53-4b73-9af8-2fa54d148b78" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687416 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ffd442-9b53-4b73-9af8-2fa54d148b78" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687428 4968 cpu_manager.go:414] "containerMap: removing container" podUID="d3240528-e402-43de-9702-45c2c308588f" containerName="revoke-keys" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687432 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3240528-e402-43de-9702-45c2c308588f" containerName="revoke-keys" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687439 4968 cpu_manager.go:414] "containerMap: removing container" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687444 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687451 4968 cpu_manager.go:414] "containerMap: removing container" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687457 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687466 4968 cpu_manager.go:414] "containerMap: removing container" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687470 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687480 4968 cpu_manager.go:414] "containerMap: removing container" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687484 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687491 4968 cpu_manager.go:414] "containerMap: removing container" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" Oct 30 14:28:11.687492 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687496 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687502 4968 cpu_manager.go:414] "containerMap: removing container" podUID="8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6" containerName="payload-pre-processing" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687507 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6" containerName="payload-pre-processing" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687519 4968 cpu_manager.go:414] "containerMap: removing container" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerName="istio-proxy" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687523 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerName="istio-proxy" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687533 4968 cpu_manager.go:414] "containerMap: removing container" podUID="aca3c382-c905-4be5-ae1a-d5f9336adc1d" containerName="main" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687537 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca3c382-c905-4be5-ae1a-d5f9336adc1d" containerName="main" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687596 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" containerName="istio-proxy" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687603 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="e294e23e-573e-4f62-a2b8-27a3dd0009b7" containerName="istio-proxy" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687610 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd4fc0f-9b38-4861-8869-03dc19722864" containerName="istio-proxy" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687616 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" containerName="istio-proxy" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687621 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe6f232-f757-4f7a-b60b-0fa7cb7a21f6" containerName="payload-pre-processing" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687627 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" containerName="istio-proxy" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687634 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca3c382-c905-4be5-ae1a-d5f9336adc1d" containerName="main" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687640 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fa2371-a482-470e-b019-baa5cd14b25c" containerName="revoke-keys" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687646 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687650 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9555cbc-5986-436f-8d2e-4550cbe5e49c" containerName="main" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687654 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" containerName="istio-proxy" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687659 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ffd442-9b53-4b73-9af8-2fa54d148b78" containerName="istio-proxy" Oct 30 14:28:11.688063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.687666 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3240528-e402-43de-9702-45c2c308588f" containerName="revoke-keys" Oct 30 14:28:11.692873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.692846 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:11.706004 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.705980 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 14:28:11.708356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.708333 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 14:28:11.708896 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.708882 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 14:28:11.713137 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.713115 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67c6566688-5rdmr"] Oct 30 14:28:11.724615 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.724587 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8kf\" (UniqueName: \"kubernetes.io/projected/65286fa1-bfc9-494c-b943-b0e4490089c9-kube-api-access-2h8kf\") pod \"maas-controller-67c6566688-5rdmr\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:11.724763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.724636 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-metrics-certs\") pod \"maas-controller-67c6566688-5rdmr\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:11.724864 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.724839 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-webhook-cert\") pod \"maas-controller-67c6566688-5rdmr\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:11.807037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.807005 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hqcqr\"" Oct 30 14:28:11.825662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.825629 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-webhook-cert\") pod \"maas-controller-67c6566688-5rdmr\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:11.825864 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.825779 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8kf\" (UniqueName: \"kubernetes.io/projected/65286fa1-bfc9-494c-b943-b0e4490089c9-kube-api-access-2h8kf\") pod \"maas-controller-67c6566688-5rdmr\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:11.825864 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.825814 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-metrics-certs\") pod \"maas-controller-67c6566688-5rdmr\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:11.828260 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.828235 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-metrics-certs\") pod \"maas-controller-67c6566688-5rdmr\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:11.833091 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.833066 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-webhook-cert\") pod \"maas-controller-67c6566688-5rdmr\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:11.847697 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:11.847673 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8kf\" (UniqueName: \"kubernetes.io/projected/65286fa1-bfc9-494c-b943-b0e4490089c9-kube-api-access-2h8kf\") pod \"maas-controller-67c6566688-5rdmr\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:12.007164 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:12.007062 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:12.352587 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:28:12.352520 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65286fa1_bfc9_494c_b943_b0e4490089c9.slice/crio-3cbb5c9b736b1bf3375ca616bfc1cbb3df96344ac3c58a6caa8462779eaecfa4 WatchSource:0}: Error finding container 3cbb5c9b736b1bf3375ca616bfc1cbb3df96344ac3c58a6caa8462779eaecfa4: Status 404 returned error can't find the container with id 3cbb5c9b736b1bf3375ca616bfc1cbb3df96344ac3c58a6caa8462779eaecfa4 Oct 30 14:28:12.354289 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:12.354264 4968 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:28:12.364624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:12.364599 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67c6566688-5rdmr"] Oct 30 14:28:12.520530 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:12.520486 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047af57e-0edc-42c0-bb08-e5306dddce4a" path="/var/lib/kubelet/pods/047af57e-0edc-42c0-bb08-e5306dddce4a/volumes" Oct 30 14:28:12.521194 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:12.521169 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0614bcf9-f822-4d09-acd4-4ae4b55a2461" path="/var/lib/kubelet/pods/0614bcf9-f822-4d09-acd4-4ae4b55a2461/volumes" Oct 30 14:28:12.521819 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:12.521794 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0ef2f1-6d69-45c0-bfe1-60438eeefb03" path="/var/lib/kubelet/pods/1c0ef2f1-6d69-45c0-bfe1-60438eeefb03/volumes" Oct 30 14:28:12.522830 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:12.522811 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6abc3e1-e0d0-41f4-b180-f82afe240a07" path="/var/lib/kubelet/pods/a6abc3e1-e0d0-41f4-b180-f82afe240a07/volumes" Oct 30 14:28:13.072483 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:13.072438 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67c6566688-5rdmr" event={"ID":"65286fa1-bfc9-494c-b943-b0e4490089c9","Type":"ContainerStarted","Data":"3cbb5c9b736b1bf3375ca616bfc1cbb3df96344ac3c58a6caa8462779eaecfa4"} Oct 30 14:28:17.119317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:17.119280 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67c6566688-5rdmr" event={"ID":"65286fa1-bfc9-494c-b943-b0e4490089c9","Type":"ContainerStarted","Data":"427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc"} Oct 30 14:28:17.119730 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:17.119381 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:17.160982 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:17.160937 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-67c6566688-5rdmr" podStartSLOduration=2.275047057 podStartE2EDuration="6.160924867s" totalImagesPullingTime="3.88587781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:17.159684249 +0000 UTC m=+1292.785127622" watchObservedRunningTime="2026-10-30 14:28:17.160924867 +0000 UTC m=+1292.786368240" Oct 30 14:28:20.443115 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:20.443083 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:28:28.133665 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:28.133589 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:35.371152 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.371115 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft"] Oct 30 14:28:35.371783 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.371764 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="41493a9c-2413-46f7-ae3d-1c79505299b4" containerName="istio-proxy" Oct 30 14:28:35.374893 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.374874 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:35.395743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.395703 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft"] Oct 30 14:28:35.397327 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.397300 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6n9ft\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:35.397457 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.397350 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k52kl\" (UniqueName: \"kubernetes.io/projected/e5995f7f-0331-4a1f-aefd-6c7173c05506-kube-api-access-k52kl\") pod \"maas-api-revoke-sub-85237e45-6n9ft\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:35.397526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.397470 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6n9ft\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:35.403749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.401790 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:28:35.403749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.401803 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:28:35.403749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.401803 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:28:35.403749 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.401924 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:28:35.498115 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.498074 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6n9ft\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:35.498328 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.498212 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6n9ft\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:35.498328 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.498261 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52kl\" (UniqueName: \"kubernetes.io/projected/e5995f7f-0331-4a1f-aefd-6c7173c05506-kube-api-access-k52kl\") pod \"maas-api-revoke-sub-85237e45-6n9ft\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:35.498944 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.498916 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6n9ft\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:35.500642 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.500617 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6n9ft\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:35.520557 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.520527 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52kl\" (UniqueName: \"kubernetes.io/projected/e5995f7f-0331-4a1f-aefd-6c7173c05506-kube-api-access-k52kl\") pod \"maas-api-revoke-sub-85237e45-6n9ft\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:35.691678 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:35.691583 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:36.010433 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:36.008355 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft"] Oct 30 14:28:36.011992 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:28:36.011956 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5995f7f_0331_4a1f_aefd_6c7173c05506.slice/crio-ce76fd1e26439e0a8d44a1e950ebcbf2ab6ccc1172d6a4423dfdcc2df065ea89 WatchSource:0}: Error finding container ce76fd1e26439e0a8d44a1e950ebcbf2ab6ccc1172d6a4423dfdcc2df065ea89: Status 404 returned error can't find the container with id ce76fd1e26439e0a8d44a1e950ebcbf2ab6ccc1172d6a4423dfdcc2df065ea89 Oct 30 14:28:36.264756 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:36.264645 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" event={"ID":"e5995f7f-0331-4a1f-aefd-6c7173c05506","Type":"ContainerStarted","Data":"ce76fd1e26439e0a8d44a1e950ebcbf2ab6ccc1172d6a4423dfdcc2df065ea89"} Oct 30 14:28:37.274175 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:37.274132 4968 generic.go:356] "Generic (PLEG): container finished" podID="e5995f7f-0331-4a1f-aefd-6c7173c05506" containerID="20083988a4630024a806ec29bb9e7509e566d1f2f82fcb941b22cb3f7a6a318f" exitCode=0 Oct 30 14:28:37.274656 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:37.274222 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" event={"ID":"e5995f7f-0331-4a1f-aefd-6c7173c05506","Type":"ContainerDied","Data":"20083988a4630024a806ec29bb9e7509e566d1f2f82fcb941b22cb3f7a6a318f"} Oct 30 14:28:38.423052 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:38.423025 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:38.535038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:38.534969 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5995f7f-0331-4a1f-aefd-6c7173c05506-kube-api-access-k52kl\" (UniqueName: \"kubernetes.io/projected/e5995f7f-0331-4a1f-aefd-6c7173c05506-kube-api-access-k52kl\") pod \"e5995f7f-0331-4a1f-aefd-6c7173c05506\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " Oct 30 14:28:38.535210 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:38.535151 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-xks-ca\") pod \"e5995f7f-0331-4a1f-aefd-6c7173c05506\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " Oct 30 14:28:38.535210 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:38.535194 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-openshift-ca\") pod \"e5995f7f-0331-4a1f-aefd-6c7173c05506\" (UID: \"e5995f7f-0331-4a1f-aefd-6c7173c05506\") " Oct 30 14:28:38.535624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:38.535597 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-openshift-ca" pod "e5995f7f-0331-4a1f-aefd-6c7173c05506" (UID: "e5995f7f-0331-4a1f-aefd-6c7173c05506"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:38.537120 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:38.537093 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-xks-ca" pod "e5995f7f-0331-4a1f-aefd-6c7173c05506" (UID: "e5995f7f-0331-4a1f-aefd-6c7173c05506"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:28:38.537213 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:38.537156 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5995f7f-0331-4a1f-aefd-6c7173c05506-kube-api-access-k52kl" pod "e5995f7f-0331-4a1f-aefd-6c7173c05506" (UID: "e5995f7f-0331-4a1f-aefd-6c7173c05506"). InnerVolumeSpecName "kube-api-access-k52kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:38.636141 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:38.636114 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k52kl\" (UniqueName: \"kubernetes.io/projected/e5995f7f-0331-4a1f-aefd-6c7173c05506-kube-api-access-k52kl\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:38.636141 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:38.636141 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:38.636316 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:38.636151 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5995f7f-0331-4a1f-aefd-6c7173c05506-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:39.290321 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:39.290293 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" Oct 30 14:28:39.290321 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:39.290299 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft" event={"ID":"e5995f7f-0331-4a1f-aefd-6c7173c05506","Type":"ContainerDied","Data":"ce76fd1e26439e0a8d44a1e950ebcbf2ab6ccc1172d6a4423dfdcc2df065ea89"} Oct 30 14:28:39.290524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:39.290342 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce76fd1e26439e0a8d44a1e950ebcbf2ab6ccc1172d6a4423dfdcc2df065ea89" Oct 30 14:28:45.079239 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:45.079208 4968 scope.go:122] "RemoveContainer" containerID="411074a10307049b4821e1346eef2fc1c0de4f4ef1bcf91111abeeeb49f8988c" Oct 30 14:28:45.089200 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:45.089178 4968 scope.go:122] "RemoveContainer" containerID="ead147a7aa59d4833cf6976320ce3a4e008c1176cf9ccad4071b9c75429d8db8" Oct 30 14:28:45.098246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:45.098223 4968 scope.go:122] "RemoveContainer" containerID="168b650286d672ff74a373896576f917cc44e7d78f27262be6fe91d83a738bb8" Oct 30 14:28:45.107251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:45.107231 4968 scope.go:122] "RemoveContainer" containerID="adee60ba53b36842cdc4e1b361eb0285c7e0a2cc7981eaef63172c4ac3899995" Oct 30 14:28:45.115667 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:45.115647 4968 scope.go:122] "RemoveContainer" containerID="0f497c55a555b893d6bf6033306edc65ac3c780213e89295958e942a16252da4" Oct 30 14:28:45.123960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:45.123941 4968 scope.go:122] "RemoveContainer" containerID="9a5033521a1c1cab8af0b57811ff9070b47991ebcf82670bc990e48324ade7b7" Oct 30 14:28:45.131997 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:45.131974 4968 scope.go:122] "RemoveContainer" containerID="1423985132e0f0a7b39a26a99ff513c13842a5765a84be1527efe7fd2681d29a" Oct 30 14:28:45.140289 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:45.140267 4968 scope.go:122] "RemoveContainer" containerID="b319d5cab1930070eacd0343e808ad1d9232b0cc444e1df42a165d10b7e1b2a4" Oct 30 14:28:45.149517 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:45.149498 4968 scope.go:122] "RemoveContainer" containerID="56c62654f40ccc4daf4f779664619928ef86ea0ba8d5e1e32446754885fa0748" Oct 30 14:28:45.157931 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:45.157914 4968 scope.go:122] "RemoveContainer" containerID="87839ac780134bd9cf8040e9e7ffd547f8a3739c40d39e3025ac02faf466e123" Oct 30 14:28:47.034874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.034838 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67c6566688-5rdmr"] Oct 30 14:28:47.035786 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.035703 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-67c6566688-5rdmr" podUID="65286fa1-bfc9-494c-b943-b0e4490089c9" containerName="manager" containerID="cri-o://427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc" gracePeriod=30 Oct 30 14:28:47.288874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.288816 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:47.357340 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.357304 4968 generic.go:356] "Generic (PLEG): container finished" podID="65286fa1-bfc9-494c-b943-b0e4490089c9" containerID="427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc" exitCode=0 Oct 30 14:28:47.357539 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.357374 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67c6566688-5rdmr" Oct 30 14:28:47.357539 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.357392 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67c6566688-5rdmr" event={"ID":"65286fa1-bfc9-494c-b943-b0e4490089c9","Type":"ContainerDied","Data":"427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc"} Oct 30 14:28:47.357539 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.357459 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67c6566688-5rdmr" event={"ID":"65286fa1-bfc9-494c-b943-b0e4490089c9","Type":"ContainerDied","Data":"3cbb5c9b736b1bf3375ca616bfc1cbb3df96344ac3c58a6caa8462779eaecfa4"} Oct 30 14:28:47.357539 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.357490 4968 scope.go:122] "RemoveContainer" containerID="427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc" Oct 30 14:28:47.369844 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.369819 4968 scope.go:122] "RemoveContainer" containerID="427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc" Oct 30 14:28:47.370179 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:47.370147 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc\": container with ID starting with 427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc not found: ID does not exist" containerID="427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc" Oct 30 14:28:47.370287 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.370191 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc"} err="failed to get container status \"427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc\": rpc error: code = NotFound desc = could not find container \"427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc\": container with ID starting with 427b6092087670eaaac1426d477987bc428d7baae5dc607dd6f92a9ed8f86cfc not found: ID does not exist" Oct 30 14:28:47.414091 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.414064 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-webhook-cert\") pod \"65286fa1-bfc9-494c-b943-b0e4490089c9\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " Oct 30 14:28:47.414245 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.414102 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-metrics-certs\") pod \"65286fa1-bfc9-494c-b943-b0e4490089c9\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " Oct 30 14:28:47.414245 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.414128 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65286fa1-bfc9-494c-b943-b0e4490089c9-kube-api-access-2h8kf\" (UniqueName: \"kubernetes.io/projected/65286fa1-bfc9-494c-b943-b0e4490089c9-kube-api-access-2h8kf\") pod \"65286fa1-bfc9-494c-b943-b0e4490089c9\" (UID: \"65286fa1-bfc9-494c-b943-b0e4490089c9\") " Oct 30 14:28:47.416135 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.416110 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-webhook-cert" pod "65286fa1-bfc9-494c-b943-b0e4490089c9" (UID: "65286fa1-bfc9-494c-b943-b0e4490089c9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:28:47.416254 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.416113 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65286fa1-bfc9-494c-b943-b0e4490089c9-kube-api-access-2h8kf" pod "65286fa1-bfc9-494c-b943-b0e4490089c9" (UID: "65286fa1-bfc9-494c-b943-b0e4490089c9"). InnerVolumeSpecName "kube-api-access-2h8kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:47.416254 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.416143 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-metrics-certs" pod "65286fa1-bfc9-494c-b943-b0e4490089c9" (UID: "65286fa1-bfc9-494c-b943-b0e4490089c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:28:47.515098 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.515065 4968 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-webhook-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:47.515098 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.515093 4968 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65286fa1-bfc9-494c-b943-b0e4490089c9-metrics-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:47.515098 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.515105 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h8kf\" (UniqueName: \"kubernetes.io/projected/65286fa1-bfc9-494c-b943-b0e4490089c9-kube-api-access-2h8kf\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:28:47.701858 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.701826 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67c6566688-5rdmr"] Oct 30 14:28:47.711333 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:47.711304 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67c6566688-5rdmr"] Oct 30 14:28:48.518032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.518001 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65286fa1-bfc9-494c-b943-b0e4490089c9" path="/var/lib/kubelet/pods/65286fa1-bfc9-494c-b943-b0e4490089c9/volumes" Oct 30 14:28:48.908242 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.908209 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67c6566688-zbvbc"] Oct 30 14:28:48.908631 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.908618 4968 cpu_manager.go:414] "containerMap: removing container" podUID="e5995f7f-0331-4a1f-aefd-6c7173c05506" containerName="revoke-keys" Oct 30 14:28:48.908681 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.908632 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5995f7f-0331-4a1f-aefd-6c7173c05506" containerName="revoke-keys" Oct 30 14:28:48.908681 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.908652 4968 cpu_manager.go:414] "containerMap: removing container" podUID="65286fa1-bfc9-494c-b943-b0e4490089c9" containerName="manager" Oct 30 14:28:48.908681 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.908657 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65286fa1-bfc9-494c-b943-b0e4490089c9" containerName="manager" Oct 30 14:28:48.908785 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.908738 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="65286fa1-bfc9-494c-b943-b0e4490089c9" containerName="manager" Oct 30 14:28:48.908785 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.908752 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5995f7f-0331-4a1f-aefd-6c7173c05506" containerName="revoke-keys" Oct 30 14:28:48.913408 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.913387 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:48.929810 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.929792 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 14:28:48.940349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.940325 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67c6566688-zbvbc"] Oct 30 14:28:48.945890 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.945868 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hqcqr\"" Oct 30 14:28:48.946008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.945868 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 14:28:48.946008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:48.945870 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 14:28:49.030146 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:49.030122 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f1d457-085f-4d3f-b3fb-f64167c5ac3c-metrics-certs\") pod \"maas-controller-67c6566688-zbvbc\" (UID: \"d3f1d457-085f-4d3f-b3fb-f64167c5ac3c\") " pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:49.030302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:49.030179 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d3f1d457-085f-4d3f-b3fb-f64167c5ac3c-webhook-cert\") pod \"maas-controller-67c6566688-zbvbc\" (UID: \"d3f1d457-085f-4d3f-b3fb-f64167c5ac3c\") " pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:49.030370 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:49.030316 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8g2\" (UniqueName: \"kubernetes.io/projected/d3f1d457-085f-4d3f-b3fb-f64167c5ac3c-kube-api-access-fx8g2\") pod \"maas-controller-67c6566688-zbvbc\" (UID: \"d3f1d457-085f-4d3f-b3fb-f64167c5ac3c\") " pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:49.131612 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:49.131510 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d3f1d457-085f-4d3f-b3fb-f64167c5ac3c-webhook-cert\") pod \"maas-controller-67c6566688-zbvbc\" (UID: \"d3f1d457-085f-4d3f-b3fb-f64167c5ac3c\") " pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:49.131778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:49.131695 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8g2\" (UniqueName: \"kubernetes.io/projected/d3f1d457-085f-4d3f-b3fb-f64167c5ac3c-kube-api-access-fx8g2\") pod \"maas-controller-67c6566688-zbvbc\" (UID: \"d3f1d457-085f-4d3f-b3fb-f64167c5ac3c\") " pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:49.131830 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:49.131786 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f1d457-085f-4d3f-b3fb-f64167c5ac3c-metrics-certs\") pod \"maas-controller-67c6566688-zbvbc\" (UID: \"d3f1d457-085f-4d3f-b3fb-f64167c5ac3c\") " pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:49.134018 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:49.133991 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d3f1d457-085f-4d3f-b3fb-f64167c5ac3c-webhook-cert\") pod \"maas-controller-67c6566688-zbvbc\" (UID: \"d3f1d457-085f-4d3f-b3fb-f64167c5ac3c\") " pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:49.134123 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:49.134034 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f1d457-085f-4d3f-b3fb-f64167c5ac3c-metrics-certs\") pod \"maas-controller-67c6566688-zbvbc\" (UID: \"d3f1d457-085f-4d3f-b3fb-f64167c5ac3c\") " pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:49.153184 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:49.153161 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8g2\" (UniqueName: \"kubernetes.io/projected/d3f1d457-085f-4d3f-b3fb-f64167c5ac3c-kube-api-access-fx8g2\") pod \"maas-controller-67c6566688-zbvbc\" (UID: \"d3f1d457-085f-4d3f-b3fb-f64167c5ac3c\") " pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:49.225817 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:49.225743 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:49.531921 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:49.531896 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67c6566688-zbvbc"] Oct 30 14:28:49.533949 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:28:49.533926 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f1d457_085f_4d3f_b3fb_f64167c5ac3c.slice/crio-1f47505d0fd81b475d2e06c0e2c8dfed6e9c072157f715c9eb2116b7a8e37fa3 WatchSource:0}: Error finding container 1f47505d0fd81b475d2e06c0e2c8dfed6e9c072157f715c9eb2116b7a8e37fa3: Status 404 returned error can't find the container with id 1f47505d0fd81b475d2e06c0e2c8dfed6e9c072157f715c9eb2116b7a8e37fa3 Oct 30 14:28:50.381414 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:50.381370 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67c6566688-zbvbc" event={"ID":"d3f1d457-085f-4d3f-b3fb-f64167c5ac3c","Type":"ContainerStarted","Data":"ae303969ee0000c8085bb8bd6bd2577690a986ee88a4a7d5999891e010dffc39"} Oct 30 14:28:50.381609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:50.381434 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67c6566688-zbvbc" event={"ID":"d3f1d457-085f-4d3f-b3fb-f64167c5ac3c","Type":"ContainerStarted","Data":"1f47505d0fd81b475d2e06c0e2c8dfed6e9c072157f715c9eb2116b7a8e37fa3"} Oct 30 14:28:50.381609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:50.381460 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:28:50.424961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:28:50.424914 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-67c6566688-zbvbc" podStartSLOduration=2.093322541 podStartE2EDuration="2.424899581s" totalImagesPullingTime="331.57704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:50.421230641 +0000 UTC m=+1326.046674014" watchObservedRunningTime="2026-10-30 14:28:50.424899581 +0000 UTC m=+1326.050342961" Oct 30 14:28:50.444544 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:28:50.444512 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:29:01.398578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:01.398537 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67c6566688-zbvbc" Oct 30 14:29:19.350634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:19.350592 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:29:20.446613 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:29:20.446563 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:29:21.758745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:21.753980 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:29:21.758745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:21.754056 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" podUID="ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:29:25.200693 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.200648 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw"] Oct 30 14:29:25.205042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.205019 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:25.226932 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.226902 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw"] Oct 30 14:29:25.232220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.232196 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:29:25.232220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.232199 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:29:25.243852 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.243821 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:29:25.244172 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.244154 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:29:25.303258 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.303228 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9lbjw\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:25.303405 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.303280 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqmnr\" (UniqueName: \"kubernetes.io/projected/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-kube-api-access-cqmnr\") pod \"maas-api-revoke-sub-c1e271a9-9lbjw\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:25.303453 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.303408 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9lbjw\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:25.404025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.403983 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqmnr\" (UniqueName: \"kubernetes.io/projected/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-kube-api-access-cqmnr\") pod \"maas-api-revoke-sub-c1e271a9-9lbjw\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:25.404185 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.404053 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9lbjw\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:25.404185 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.404125 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9lbjw\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:25.404760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.404732 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9lbjw\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:25.406368 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.406349 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9lbjw\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:25.425584 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.425560 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqmnr\" (UniqueName: \"kubernetes.io/projected/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-kube-api-access-cqmnr\") pod \"maas-api-revoke-sub-c1e271a9-9lbjw\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:25.518954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.518882 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:25.824485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:25.822319 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw"] Oct 30 14:29:25.828461 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:29:25.828432 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfca8db_ef2e_46c1_ba78_b787ddff44c9.slice/crio-1c20389575085e476aa93603e32549fe28830cd634844502870199c7f51a15e9 WatchSource:0}: Error finding container 1c20389575085e476aa93603e32549fe28830cd634844502870199c7f51a15e9: Status 404 returned error can't find the container with id 1c20389575085e476aa93603e32549fe28830cd634844502870199c7f51a15e9 Oct 30 14:29:26.669044 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:26.669014 4968 generic.go:356] "Generic (PLEG): container finished" podID="7cfca8db-ef2e-46c1-ba78-b787ddff44c9" containerID="d8b18a01d6afcd002627c8f4727c05a0ff2f79e1b7032e19fee60961c97fda95" exitCode=0 Oct 30 14:29:26.669433 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:26.669093 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" event={"ID":"7cfca8db-ef2e-46c1-ba78-b787ddff44c9","Type":"ContainerDied","Data":"d8b18a01d6afcd002627c8f4727c05a0ff2f79e1b7032e19fee60961c97fda95"} Oct 30 14:29:26.669433 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:26.669159 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" event={"ID":"7cfca8db-ef2e-46c1-ba78-b787ddff44c9","Type":"ContainerStarted","Data":"1c20389575085e476aa93603e32549fe28830cd634844502870199c7f51a15e9"} Oct 30 14:29:27.813779 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:27.813757 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:27.823900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:27.823876 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-xks-ca\") pod \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " Oct 30 14:29:27.823994 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:27.823979 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-kube-api-access-cqmnr\" (UniqueName: \"kubernetes.io/projected/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-kube-api-access-cqmnr\") pod \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " Oct 30 14:29:27.824043 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:27.824026 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-openshift-ca\") pod \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\" (UID: \"7cfca8db-ef2e-46c1-ba78-b787ddff44c9\") " Oct 30 14:29:27.824395 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:27.824368 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-openshift-ca" pod "7cfca8db-ef2e-46c1-ba78-b787ddff44c9" (UID: "7cfca8db-ef2e-46c1-ba78-b787ddff44c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:29:27.825848 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:27.825825 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-xks-ca" pod "7cfca8db-ef2e-46c1-ba78-b787ddff44c9" (UID: "7cfca8db-ef2e-46c1-ba78-b787ddff44c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:29:27.825982 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:27.825965 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-kube-api-access-cqmnr" pod "7cfca8db-ef2e-46c1-ba78-b787ddff44c9" (UID: "7cfca8db-ef2e-46c1-ba78-b787ddff44c9"). InnerVolumeSpecName "kube-api-access-cqmnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:29:27.925343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:27.925270 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:29:27.925343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:27.925294 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:29:27.925343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:27.925304 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqmnr\" (UniqueName: \"kubernetes.io/projected/7cfca8db-ef2e-46c1-ba78-b787ddff44c9-kube-api-access-cqmnr\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:29:28.683212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:28.683183 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" Oct 30 14:29:28.683212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:28.683191 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw" event={"ID":"7cfca8db-ef2e-46c1-ba78-b787ddff44c9","Type":"ContainerDied","Data":"1c20389575085e476aa93603e32549fe28830cd634844502870199c7f51a15e9"} Oct 30 14:29:28.683455 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:28.683236 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c20389575085e476aa93603e32549fe28830cd634844502870199c7f51a15e9" Oct 30 14:29:30.033160 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:30.033120 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv"] Oct 30 14:29:30.042536 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:30.042512 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e55eaf-wnrcv"] Oct 30 14:29:30.518129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:30.518096 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0e3219-8050-48de-98b3-0962851a502b" path="/var/lib/kubelet/pods/9e0e3219-8050-48de-98b3-0962851a502b/volumes" Oct 30 14:29:31.034309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:31.034264 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk"] Oct 30 14:29:31.045034 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:31.044997 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b5696b-7fqbk"] Oct 30 14:29:32.517976 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:32.517939 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14f6418-d532-46fb-aabe-988b8516284c" path="/var/lib/kubelet/pods/d14f6418-d532-46fb-aabe-988b8516284c/volumes" Oct 30 14:29:36.480009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:36.479970 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:29:40.982213 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:40.982169 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:29:43.042829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:43.042764 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx"] Oct 30 14:29:43.051852 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:43.051816 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc029af9-h5qpx"] Oct 30 14:29:44.031924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:44.031886 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm"] Oct 30 14:29:44.041882 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:44.041842 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739e8215-h5tvm"] Oct 30 14:29:44.518259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:44.518225 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e55be46-5a85-4a9d-a82e-587bef35635b" path="/var/lib/kubelet/pods/5e55be46-5a85-4a9d-a82e-587bef35635b/volumes" Oct 30 14:29:44.518696 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:44.518641 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c665c064-6a70-4808-9c35-cbf48f29f0b7" path="/var/lib/kubelet/pods/c665c064-6a70-4808-9c35-cbf48f29f0b7/volumes" Oct 30 14:29:45.021953 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.021922 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7"] Oct 30 14:29:45.022358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.022346 4968 cpu_manager.go:414] "containerMap: removing container" podUID="7cfca8db-ef2e-46c1-ba78-b787ddff44c9" containerName="revoke-keys" Oct 30 14:29:45.022403 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.022359 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfca8db-ef2e-46c1-ba78-b787ddff44c9" containerName="revoke-keys" Oct 30 14:29:45.022456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.022447 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfca8db-ef2e-46c1-ba78-b787ddff44c9" containerName="revoke-keys" Oct 30 14:29:45.025408 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.025393 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:45.039008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.038985 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:29:45.039008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.038999 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:29:45.039008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.039000 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:29:45.039194 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.039021 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:29:45.046261 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.046236 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7"] Oct 30 14:29:45.088168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.088136 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b77p7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:45.088316 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.088181 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b77p7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:45.088316 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.088216 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfzlx\" (UniqueName: \"kubernetes.io/projected/b01e9983-027b-450f-bc1c-a3764d429dd7-kube-api-access-vfzlx\") pod \"maas-api-revoke-sub-2e1f5b5f-b77p7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:45.189406 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.189368 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b77p7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:45.189630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.189421 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfzlx\" (UniqueName: \"kubernetes.io/projected/b01e9983-027b-450f-bc1c-a3764d429dd7-kube-api-access-vfzlx\") pod \"maas-api-revoke-sub-2e1f5b5f-b77p7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:45.189630 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.189551 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b77p7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:45.190105 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.190082 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b77p7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:45.191917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.191893 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b77p7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:45.211208 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.211183 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfzlx\" (UniqueName: \"kubernetes.io/projected/b01e9983-027b-450f-bc1c-a3764d429dd7-kube-api-access-vfzlx\") pod \"maas-api-revoke-sub-2e1f5b5f-b77p7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:45.275846 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.275769 4968 scope.go:122] "RemoveContainer" containerID="b74bb2867d03a936c6a76d9ab696055368008569b405cd3d3d7037a2d16bef88" Oct 30 14:29:45.285858 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.285838 4968 scope.go:122] "RemoveContainer" containerID="2fddbbfe21c8b8b88805c5787ce333ad91cef6c25e20d37a86b013113e3a8913" Oct 30 14:29:45.294609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.294593 4968 scope.go:122] "RemoveContainer" containerID="7f30f352d0efb187c0a41a2fa726a5ffd0712dae8cefaa84f46327dd8cd59f76" Oct 30 14:29:45.304222 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.304201 4968 scope.go:122] "RemoveContainer" containerID="0bcd9a49c4ecf84e75269057e889b08412912654c2695c3df273b018c88d988b" Oct 30 14:29:45.341860 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.341838 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:45.648101 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.648060 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7"] Oct 30 14:29:45.653232 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:29:45.653204 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01e9983_027b_450f_bc1c_a3764d429dd7.slice/crio-42d062c84b89fbe920affc72ce98631e7fddde2b7c0dd6d6852a32cba720d907 WatchSource:0}: Error finding container 42d062c84b89fbe920affc72ce98631e7fddde2b7c0dd6d6852a32cba720d907: Status 404 returned error can't find the container with id 42d062c84b89fbe920affc72ce98631e7fddde2b7c0dd6d6852a32cba720d907 Oct 30 14:29:45.836956 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:45.836924 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" event={"ID":"b01e9983-027b-450f-bc1c-a3764d429dd7","Type":"ContainerStarted","Data":"42d062c84b89fbe920affc72ce98631e7fddde2b7c0dd6d6852a32cba720d907"} Oct 30 14:29:46.845670 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:46.845637 4968 generic.go:356] "Generic (PLEG): container finished" podID="b01e9983-027b-450f-bc1c-a3764d429dd7" containerID="ada87fa51b793f8dbacf3fe18333c18239772b45a6d638dffcc1434ab6b52745" exitCode=0 Oct 30 14:29:46.846080 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:46.845739 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" event={"ID":"b01e9983-027b-450f-bc1c-a3764d429dd7","Type":"ContainerDied","Data":"ada87fa51b793f8dbacf3fe18333c18239772b45a6d638dffcc1434ab6b52745"} Oct 30 14:29:47.999110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:47.999077 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:48.120450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.120368 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01e9983-027b-450f-bc1c-a3764d429dd7-kube-api-access-vfzlx\" (UniqueName: \"kubernetes.io/projected/b01e9983-027b-450f-bc1c-a3764d429dd7-kube-api-access-vfzlx\") pod \"b01e9983-027b-450f-bc1c-a3764d429dd7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " Oct 30 14:29:48.120450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.120410 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-openshift-ca\") pod \"b01e9983-027b-450f-bc1c-a3764d429dd7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " Oct 30 14:29:48.120631 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.120493 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-xks-ca\") pod \"b01e9983-027b-450f-bc1c-a3764d429dd7\" (UID: \"b01e9983-027b-450f-bc1c-a3764d429dd7\") " Oct 30 14:29:48.120838 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.120813 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-openshift-ca" pod "b01e9983-027b-450f-bc1c-a3764d429dd7" (UID: "b01e9983-027b-450f-bc1c-a3764d429dd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:29:48.122578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.122549 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-xks-ca" pod "b01e9983-027b-450f-bc1c-a3764d429dd7" (UID: "b01e9983-027b-450f-bc1c-a3764d429dd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:29:48.122578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.122552 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01e9983-027b-450f-bc1c-a3764d429dd7-kube-api-access-vfzlx" pod "b01e9983-027b-450f-bc1c-a3764d429dd7" (UID: "b01e9983-027b-450f-bc1c-a3764d429dd7"). InnerVolumeSpecName "kube-api-access-vfzlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:29:48.221520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.221488 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:29:48.221520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.221517 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfzlx\" (UniqueName: \"kubernetes.io/projected/b01e9983-027b-450f-bc1c-a3764d429dd7-kube-api-access-vfzlx\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:29:48.221520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.221527 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01e9983-027b-450f-bc1c-a3764d429dd7-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:29:48.861798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.861759 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" event={"ID":"b01e9983-027b-450f-bc1c-a3764d429dd7","Type":"ContainerDied","Data":"42d062c84b89fbe920affc72ce98631e7fddde2b7c0dd6d6852a32cba720d907"} Oct 30 14:29:48.861967 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.861807 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42d062c84b89fbe920affc72ce98631e7fddde2b7c0dd6d6852a32cba720d907" Oct 30 14:29:48.861967 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:48.861776 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7" Oct 30 14:29:49.031598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:49.031563 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd"] Oct 30 14:29:49.036186 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:49.036158 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe8a064-tjrvd"] Oct 30 14:29:50.445179 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:29:50.445145 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:29:50.518514 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:50.518462 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd35601-9586-49af-9064-29963233e94c" path="/var/lib/kubelet/pods/bfd35601-9586-49af-9064-29963233e94c/volumes" Oct 30 14:29:56.242180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.242109 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx"] Oct 30 14:29:56.242556 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.242536 4968 cpu_manager.go:414] "containerMap: removing container" podUID="b01e9983-027b-450f-bc1c-a3764d429dd7" containerName="revoke-keys" Oct 30 14:29:56.242556 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.242551 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01e9983-027b-450f-bc1c-a3764d429dd7" containerName="revoke-keys" Oct 30 14:29:56.242664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.242653 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01e9983-027b-450f-bc1c-a3764d429dd7" containerName="revoke-keys" Oct 30 14:29:56.246060 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.246041 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:56.268838 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.268812 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:29:56.269079 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.269057 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:29:56.269317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.269173 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:29:56.269317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.269223 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:29:56.274237 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.274214 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx"] Oct 30 14:29:56.300769 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.300735 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9rqnx\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:56.300909 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.300829 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r82vf\" (UniqueName: \"kubernetes.io/projected/c2589b6b-d190-44d1-a68b-329ecd3cc941-kube-api-access-r82vf\") pod \"maas-api-revoke-sub-ec843018-9rqnx\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:56.301002 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.300983 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9rqnx\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:56.401634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.401598 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9rqnx\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:56.401833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.401656 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9rqnx\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:56.401833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.401679 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r82vf\" (UniqueName: \"kubernetes.io/projected/c2589b6b-d190-44d1-a68b-329ecd3cc941-kube-api-access-r82vf\") pod \"maas-api-revoke-sub-ec843018-9rqnx\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:56.402349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.402321 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9rqnx\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:56.403980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.403959 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9rqnx\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:56.425489 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.425467 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r82vf\" (UniqueName: \"kubernetes.io/projected/c2589b6b-d190-44d1-a68b-329ecd3cc941-kube-api-access-r82vf\") pod \"maas-api-revoke-sub-ec843018-9rqnx\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:56.577129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.577047 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:56.881513 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.881489 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx"] Oct 30 14:29:56.883267 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:29:56.883241 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2589b6b_d190_44d1_a68b_329ecd3cc941.slice/crio-6db51059cb2d32ce6573f020580f4551fca3dbc806921b515301c765dbaeb470 WatchSource:0}: Error finding container 6db51059cb2d32ce6573f020580f4551fca3dbc806921b515301c765dbaeb470: Status 404 returned error can't find the container with id 6db51059cb2d32ce6573f020580f4551fca3dbc806921b515301c765dbaeb470 Oct 30 14:29:56.912201 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:56.912168 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" event={"ID":"c2589b6b-d190-44d1-a68b-329ecd3cc941","Type":"ContainerStarted","Data":"6db51059cb2d32ce6573f020580f4551fca3dbc806921b515301c765dbaeb470"} Oct 30 14:29:57.921634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:57.921601 4968 generic.go:356] "Generic (PLEG): container finished" podID="c2589b6b-d190-44d1-a68b-329ecd3cc941" containerID="9be5edd330de55aa84f0c8e3b6cf80a6e995fddf9bcc23f018d3d875ead44190" exitCode=0 Oct 30 14:29:57.922016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:57.921681 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" event={"ID":"c2589b6b-d190-44d1-a68b-329ecd3cc941","Type":"ContainerDied","Data":"9be5edd330de55aa84f0c8e3b6cf80a6e995fddf9bcc23f018d3d875ead44190"} Oct 30 14:29:57.956497 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:57.956469 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:29:59.033992 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.033962 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7"] Oct 30 14:29:59.064484 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.064449 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f779cb64-zjdn7"] Oct 30 14:29:59.090276 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.090253 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:59.240322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.240237 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2589b6b-d190-44d1-a68b-329ecd3cc941-kube-api-access-r82vf\" (UniqueName: \"kubernetes.io/projected/c2589b6b-d190-44d1-a68b-329ecd3cc941-kube-api-access-r82vf\") pod \"c2589b6b-d190-44d1-a68b-329ecd3cc941\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " Oct 30 14:29:59.240322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.240299 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-xks-ca\") pod \"c2589b6b-d190-44d1-a68b-329ecd3cc941\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " Oct 30 14:29:59.240565 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.240401 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-openshift-ca\") pod \"c2589b6b-d190-44d1-a68b-329ecd3cc941\" (UID: \"c2589b6b-d190-44d1-a68b-329ecd3cc941\") " Oct 30 14:29:59.240758 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.240710 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-openshift-ca" pod "c2589b6b-d190-44d1-a68b-329ecd3cc941" (UID: "c2589b6b-d190-44d1-a68b-329ecd3cc941"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:29:59.242422 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.242396 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2589b6b-d190-44d1-a68b-329ecd3cc941-kube-api-access-r82vf" pod "c2589b6b-d190-44d1-a68b-329ecd3cc941" (UID: "c2589b6b-d190-44d1-a68b-329ecd3cc941"). InnerVolumeSpecName "kube-api-access-r82vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:29:59.242422 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.242407 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-xks-ca" pod "c2589b6b-d190-44d1-a68b-329ecd3cc941" (UID: "c2589b6b-d190-44d1-a68b-329ecd3cc941"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:29:59.342113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.342080 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:29:59.342113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.342109 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r82vf\" (UniqueName: \"kubernetes.io/projected/c2589b6b-d190-44d1-a68b-329ecd3cc941-kube-api-access-r82vf\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:29:59.342113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.342120 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2589b6b-d190-44d1-a68b-329ecd3cc941-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:29:59.935827 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.935799 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" Oct 30 14:29:59.935984 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.935926 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx" event={"ID":"c2589b6b-d190-44d1-a68b-329ecd3cc941","Type":"ContainerDied","Data":"6db51059cb2d32ce6573f020580f4551fca3dbc806921b515301c765dbaeb470"} Oct 30 14:29:59.935984 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:29:59.935967 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db51059cb2d32ce6573f020580f4551fca3dbc806921b515301c765dbaeb470" Oct 30 14:30:00.152607 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.152575 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd"] Oct 30 14:30:00.153013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.152997 4968 cpu_manager.go:414] "containerMap: removing container" podUID="c2589b6b-d190-44d1-a68b-329ecd3cc941" containerName="revoke-keys" Oct 30 14:30:00.153013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.153009 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2589b6b-d190-44d1-a68b-329ecd3cc941" containerName="revoke-keys" Oct 30 14:30:00.153114 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.153103 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2589b6b-d190-44d1-a68b-329ecd3cc941" containerName="revoke-keys" Oct 30 14:30:00.156941 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.156914 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" Oct 30 14:30:00.172887 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.172863 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:30:00.174903 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.174881 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:30:00.179147 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.179125 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd"] Oct 30 14:30:00.182425 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.182408 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:30:00.249023 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.248928 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889510-29czd\" (UID: \"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" Oct 30 14:30:00.249023 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.249017 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q55s\" (UniqueName: \"kubernetes.io/projected/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-kube-api-access-9q55s\") pod \"maas-api-key-cleanup-29889510-29czd\" (UID: \"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" Oct 30 14:30:00.350287 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.350238 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q55s\" (UniqueName: \"kubernetes.io/projected/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-kube-api-access-9q55s\") pod \"maas-api-key-cleanup-29889510-29czd\" (UID: \"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" Oct 30 14:30:00.350446 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.350358 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889510-29czd\" (UID: \"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" Oct 30 14:30:00.350975 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.350957 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889510-29czd\" (UID: \"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" Oct 30 14:30:00.372658 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.372629 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q55s\" (UniqueName: \"kubernetes.io/projected/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-kube-api-access-9q55s\") pod \"maas-api-key-cleanup-29889510-29czd\" (UID: \"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" Oct 30 14:30:00.470356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.470326 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" Oct 30 14:30:00.520397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.520319 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec31a55-ad49-4fc6-8e6a-32205bceec8b" path="/var/lib/kubelet/pods/4ec31a55-ad49-4fc6-8e6a-32205bceec8b/volumes" Oct 30 14:30:00.783309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.783286 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd"] Oct 30 14:30:00.786017 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:30:00.785987 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5391c0_b2da_4a48_b23d_8d10c28d7f5f.slice/crio-b7a514af6c85bbd76fbc6b07c23b2134d8d1a0c6daf9e5c98032d93bd0057dbc WatchSource:0}: Error finding container b7a514af6c85bbd76fbc6b07c23b2134d8d1a0c6daf9e5c98032d93bd0057dbc: Status 404 returned error can't find the container with id b7a514af6c85bbd76fbc6b07c23b2134d8d1a0c6daf9e5c98032d93bd0057dbc Oct 30 14:30:00.943653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:00.943623 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" event={"ID":"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f","Type":"ContainerStarted","Data":"b7a514af6c85bbd76fbc6b07c23b2134d8d1a0c6daf9e5c98032d93bd0057dbc"} Oct 30 14:30:01.951574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:01.951475 4968 generic.go:356] "Generic (PLEG): container finished" podID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerID="7eee51f4cc4ee8a8d7daad91d8824a30e4485573b6debe42caf15b299571421a" exitCode=60 Oct 30 14:30:01.951574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:01.951519 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" event={"ID":"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f","Type":"ContainerDied","Data":"7eee51f4cc4ee8a8d7daad91d8824a30e4485573b6debe42caf15b299571421a"} Oct 30 14:30:01.952125 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:01.951799 4968 scope.go:122] "RemoveContainer" containerID="7eee51f4cc4ee8a8d7daad91d8824a30e4485573b6debe42caf15b299571421a" Oct 30 14:30:02.038847 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:02.038809 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb"] Oct 30 14:30:02.045597 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:02.045568 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd59e5c6-xzsjb"] Oct 30 14:30:02.518457 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:02.518418 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bbba234-6f5d-46ba-beae-72844a22252e" path="/var/lib/kubelet/pods/3bbba234-6f5d-46ba-beae-72844a22252e/volumes" Oct 30 14:30:02.960002 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:02.959969 4968 generic.go:356] "Generic (PLEG): container finished" podID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerID="b4344b41dd6d25f0f8debf9efd60e214e276e706d1a798c632994dcbbc1e2536" exitCode=60 Oct 30 14:30:02.960431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:02.960058 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" event={"ID":"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f","Type":"ContainerDied","Data":"b4344b41dd6d25f0f8debf9efd60e214e276e706d1a798c632994dcbbc1e2536"} Oct 30 14:30:02.960431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:02.960118 4968 scope.go:122] "RemoveContainer" containerID="7eee51f4cc4ee8a8d7daad91d8824a30e4485573b6debe42caf15b299571421a" Oct 30 14:30:02.960431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:02.960279 4968 scope.go:122] "RemoveContainer" containerID="b4344b41dd6d25f0f8debf9efd60e214e276e706d1a798c632994dcbbc1e2536" Oct 30 14:30:02.960597 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:30:02.960576 4968 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889510-29czd_odh-ai-gateway-infra(2a5391c0-b2da-4a48-b23d-8d10c28d7f5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" Oct 30 14:30:03.968667 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:03.968641 4968 scope.go:122] "RemoveContainer" containerID="b4344b41dd6d25f0f8debf9efd60e214e276e706d1a798c632994dcbbc1e2536" Oct 30 14:30:03.969174 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:30:03.968876 4968 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889510-29czd_odh-ai-gateway-infra(2a5391c0-b2da-4a48-b23d-8d10c28d7f5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" Oct 30 14:30:04.030505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:04.030474 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv"] Oct 30 14:30:04.033227 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:04.033201 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa00f5b9-6rnkv"] Oct 30 14:30:04.517913 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:04.517881 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afef9040-cff0-40c0-b50a-9f0aea368e8d" path="/var/lib/kubelet/pods/afef9040-cff0-40c0-b50a-9f0aea368e8d/volumes" Oct 30 14:30:11.886603 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:11.886564 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:30:12.038742 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:12.038700 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq"] Oct 30 14:30:12.048437 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:12.048412 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdffe94f-cn2bq"] Oct 30 14:30:12.520880 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:12.520838 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99216a44-0d65-4f90-8d7b-5ee32b038d52" path="/var/lib/kubelet/pods/99216a44-0d65-4f90-8d7b-5ee32b038d52/volumes" Oct 30 14:30:14.540854 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:14.540822 4968 scope.go:122] "RemoveContainer" containerID="b4344b41dd6d25f0f8debf9efd60e214e276e706d1a798c632994dcbbc1e2536" Oct 30 14:30:15.060053 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:15.060016 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" event={"ID":"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f","Type":"ContainerStarted","Data":"d0c336791676ccd5acae0e1171b27434b7640363133eb2e53c65c5ff4af817e9"} Oct 30 14:30:15.099082 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:15.099036 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" podStartSLOduration=15.09902088 podStartE2EDuration="15.09902088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:15.096353147 +0000 UTC m=+1410.721796530" watchObservedRunningTime="2026-10-30 14:30:15.09902088 +0000 UTC m=+1410.724464279" Oct 30 14:30:16.068292 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:16.068257 4968 generic.go:356] "Generic (PLEG): container finished" podID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerID="d0c336791676ccd5acae0e1171b27434b7640363133eb2e53c65c5ff4af817e9" exitCode=60 Oct 30 14:30:16.068736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:16.068329 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" event={"ID":"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f","Type":"ContainerDied","Data":"d0c336791676ccd5acae0e1171b27434b7640363133eb2e53c65c5ff4af817e9"} Oct 30 14:30:16.068736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:16.068389 4968 scope.go:122] "RemoveContainer" containerID="b4344b41dd6d25f0f8debf9efd60e214e276e706d1a798c632994dcbbc1e2536" Oct 30 14:30:16.068736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:16.068667 4968 scope.go:122] "RemoveContainer" containerID="d0c336791676ccd5acae0e1171b27434b7640363133eb2e53c65c5ff4af817e9" Oct 30 14:30:16.068941 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:30:16.068904 4968 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889510-29czd_odh-ai-gateway-infra(2a5391c0-b2da-4a48-b23d-8d10c28d7f5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" Oct 30 14:30:16.116743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:16.116702 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd"] Oct 30 14:30:17.031776 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:17.031738 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng"] Oct 30 14:30:17.041127 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:17.041100 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefc604d-sk9ng"] Oct 30 14:30:17.216782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:17.216758 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" Oct 30 14:30:17.277793 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:17.277761 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-kube-api-access-9q55s\" (UniqueName: \"kubernetes.io/projected/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-kube-api-access-9q55s\") pod \"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f\" (UID: \"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f\") " Oct 30 14:30:17.277938 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:17.277848 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-maas-api-service-ca\") pod \"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f\" (UID: \"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f\") " Oct 30 14:30:17.278204 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:17.278180 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-maas-api-service-ca" pod "2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" (UID: "2a5391c0-b2da-4a48-b23d-8d10c28d7f5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:17.279924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:17.279898 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-kube-api-access-9q55s" pod "2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" (UID: "2a5391c0-b2da-4a48-b23d-8d10c28d7f5f"). InnerVolumeSpecName "kube-api-access-9q55s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:17.378617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:17.378582 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q55s\" (UniqueName: \"kubernetes.io/projected/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-kube-api-access-9q55s\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:30:17.378617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:17.378616 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:30:18.085332 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:18.085304 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" Oct 30 14:30:18.085541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:18.085300 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd" event={"ID":"2a5391c0-b2da-4a48-b23d-8d10c28d7f5f","Type":"ContainerDied","Data":"b7a514af6c85bbd76fbc6b07c23b2134d8d1a0c6daf9e5c98032d93bd0057dbc"} Oct 30 14:30:18.085541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:18.085459 4968 scope.go:122] "RemoveContainer" containerID="d0c336791676ccd5acae0e1171b27434b7640363133eb2e53c65c5ff4af817e9" Oct 30 14:30:18.132015 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:18.131979 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd"] Oct 30 14:30:18.141990 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:18.141962 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-29czd"] Oct 30 14:30:18.519049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:18.518971 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" path="/var/lib/kubelet/pods/2a5391c0-b2da-4a48-b23d-8d10c28d7f5f/volumes" Oct 30 14:30:18.519415 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:18.519392 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d762a3-661e-4b35-a6b4-14b60f966066" path="/var/lib/kubelet/pods/d6d762a3-661e-4b35-a6b4-14b60f966066/volumes" Oct 30 14:30:20.443825 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:30:20.443790 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:30:22.034180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:22.034136 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h"] Oct 30 14:30:22.043709 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:22.043680 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde66139-2x72h"] Oct 30 14:30:22.518123 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:22.518088 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27fd7342-ae8c-4515-af80-7df58976e7e6" path="/var/lib/kubelet/pods/27fd7342-ae8c-4515-af80-7df58976e7e6/volumes" Oct 30 14:30:23.974883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:23.974850 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx"] Oct 30 14:30:23.975338 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:23.975324 4968 cpu_manager.go:414] "containerMap: removing container" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerName="cleanup" Oct 30 14:30:23.975338 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:23.975339 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerName="cleanup" Oct 30 14:30:23.975420 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:23.975365 4968 cpu_manager.go:414] "containerMap: removing container" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerName="cleanup" Oct 30 14:30:23.975420 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:23.975371 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerName="cleanup" Oct 30 14:30:23.975420 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:23.975382 4968 cpu_manager.go:414] "containerMap: removing container" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerName="cleanup" Oct 30 14:30:23.975420 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:23.975387 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerName="cleanup" Oct 30 14:30:23.975545 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:23.975449 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerName="cleanup" Oct 30 14:30:23.975545 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:23.975459 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerName="cleanup" Oct 30 14:30:23.980198 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:23.980181 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:23.999669 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:23.999641 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx"] Oct 30 14:30:24.005859 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.005836 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:30:24.006010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.005866 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:30:24.006081 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.005904 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:30:24.006081 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.005921 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:30:24.032437 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.032411 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-h46nx\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:24.032601 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.032547 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-h46nx\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:24.032660 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.032627 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrf87\" (UniqueName: \"kubernetes.io/projected/d20f347e-1957-4233-9a5a-7e8a1e40410d-kube-api-access-wrf87\") pod \"maas-api-revoke-sub-046e1343-h46nx\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:24.133552 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.133523 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-h46nx\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:24.133772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.133580 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrf87\" (UniqueName: \"kubernetes.io/projected/d20f347e-1957-4233-9a5a-7e8a1e40410d-kube-api-access-wrf87\") pod \"maas-api-revoke-sub-046e1343-h46nx\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:24.133772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.133619 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-h46nx\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:24.134286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.134265 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-h46nx\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:24.136038 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.136005 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-h46nx\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:24.155672 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.155643 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrf87\" (UniqueName: \"kubernetes.io/projected/d20f347e-1957-4233-9a5a-7e8a1e40410d-kube-api-access-wrf87\") pod \"maas-api-revoke-sub-046e1343-h46nx\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:24.293271 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.293198 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:24.598638 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:24.598612 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx"] Oct 30 14:30:24.600846 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:30:24.600815 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20f347e_1957_4233_9a5a_7e8a1e40410d.slice/crio-14976e17e9acf7a2e76fb53d4b9c556efffa58939cef97ad9841c2b9a1dd6ef6 WatchSource:0}: Error finding container 14976e17e9acf7a2e76fb53d4b9c556efffa58939cef97ad9841c2b9a1dd6ef6: Status 404 returned error can't find the container with id 14976e17e9acf7a2e76fb53d4b9c556efffa58939cef97ad9841c2b9a1dd6ef6 Oct 30 14:30:25.134779 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:25.134740 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" event={"ID":"d20f347e-1957-4233-9a5a-7e8a1e40410d","Type":"ContainerStarted","Data":"99ec97a24c30332c17e39f49573bbcd81601de7a4e68836cdad582b8d1f14d81"} Oct 30 14:30:25.135174 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:25.134802 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" event={"ID":"d20f347e-1957-4233-9a5a-7e8a1e40410d","Type":"ContainerStarted","Data":"14976e17e9acf7a2e76fb53d4b9c556efffa58939cef97ad9841c2b9a1dd6ef6"} Oct 30 14:30:25.174978 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:25.174869 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" podStartSLOduration=2.174851715 podStartE2EDuration="2.174851715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:25.172738681 +0000 UTC m=+1420.798182047" watchObservedRunningTime="2026-10-30 14:30:25.174851715 +0000 UTC m=+1420.800295087" Oct 30 14:30:26.142780 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:26.142750 4968 generic.go:356] "Generic (PLEG): container finished" podID="d20f347e-1957-4233-9a5a-7e8a1e40410d" containerID="99ec97a24c30332c17e39f49573bbcd81601de7a4e68836cdad582b8d1f14d81" exitCode=0 Oct 30 14:30:26.143191 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:26.142824 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" event={"ID":"d20f347e-1957-4233-9a5a-7e8a1e40410d","Type":"ContainerDied","Data":"99ec97a24c30332c17e39f49573bbcd81601de7a4e68836cdad582b8d1f14d81"} Oct 30 14:30:27.286608 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:27.286579 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:27.366466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:27.366439 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20f347e-1957-4233-9a5a-7e8a1e40410d-kube-api-access-wrf87\" (UniqueName: \"kubernetes.io/projected/d20f347e-1957-4233-9a5a-7e8a1e40410d-kube-api-access-wrf87\") pod \"d20f347e-1957-4233-9a5a-7e8a1e40410d\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " Oct 30 14:30:27.366623 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:27.366530 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-xks-ca\") pod \"d20f347e-1957-4233-9a5a-7e8a1e40410d\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " Oct 30 14:30:27.366623 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:27.366565 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-openshift-ca\") pod \"d20f347e-1957-4233-9a5a-7e8a1e40410d\" (UID: \"d20f347e-1957-4233-9a5a-7e8a1e40410d\") " Oct 30 14:30:27.366969 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:27.366938 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-openshift-ca" pod "d20f347e-1957-4233-9a5a-7e8a1e40410d" (UID: "d20f347e-1957-4233-9a5a-7e8a1e40410d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:27.368468 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:27.368443 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20f347e-1957-4233-9a5a-7e8a1e40410d-kube-api-access-wrf87" pod "d20f347e-1957-4233-9a5a-7e8a1e40410d" (UID: "d20f347e-1957-4233-9a5a-7e8a1e40410d"). InnerVolumeSpecName "kube-api-access-wrf87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:27.368539 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:27.368509 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-xks-ca" pod "d20f347e-1957-4233-9a5a-7e8a1e40410d" (UID: "d20f347e-1957-4233-9a5a-7e8a1e40410d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:30:27.467586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:27.467517 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrf87\" (UniqueName: \"kubernetes.io/projected/d20f347e-1957-4233-9a5a-7e8a1e40410d-kube-api-access-wrf87\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:30:27.467586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:27.467542 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:30:27.467586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:27.467551 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d20f347e-1957-4233-9a5a-7e8a1e40410d-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:30:28.033690 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:28.033654 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4"] Oct 30 14:30:28.047958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:28.047920 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb268ef-s65k4"] Oct 30 14:30:28.158589 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:28.158560 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" Oct 30 14:30:28.158812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:28.158596 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx" event={"ID":"d20f347e-1957-4233-9a5a-7e8a1e40410d","Type":"ContainerDied","Data":"14976e17e9acf7a2e76fb53d4b9c556efffa58939cef97ad9841c2b9a1dd6ef6"} Oct 30 14:30:28.158812 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:28.158669 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14976e17e9acf7a2e76fb53d4b9c556efffa58939cef97ad9841c2b9a1dd6ef6" Oct 30 14:30:28.518191 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:28.518158 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5060dc40-67c7-4d61-8829-a84041f026e1" path="/var/lib/kubelet/pods/5060dc40-67c7-4d61-8829-a84041f026e1/volumes" Oct 30 14:30:29.032447 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:29.032409 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx"] Oct 30 14:30:29.042058 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:29.042030 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72adc37f-w2wfx"] Oct 30 14:30:30.517459 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:30.517424 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231675b8-2cda-4d8f-ab74-03164765f8d3" path="/var/lib/kubelet/pods/231675b8-2cda-4d8f-ab74-03164765f8d3/volumes" Oct 30 14:30:33.031899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:33.031862 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j"] Oct 30 14:30:33.041538 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:33.041509 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc44f81-bhx6j"] Oct 30 14:30:34.031112 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.031081 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk"] Oct 30 14:30:34.040261 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.040233 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c624f054-b42dk"] Oct 30 14:30:34.469075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.469040 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d"] Oct 30 14:30:34.469515 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.469503 4968 cpu_manager.go:414] "containerMap: removing container" podUID="d20f347e-1957-4233-9a5a-7e8a1e40410d" containerName="revoke-keys" Oct 30 14:30:34.469563 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.469517 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20f347e-1957-4233-9a5a-7e8a1e40410d" containerName="revoke-keys" Oct 30 14:30:34.469599 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.469587 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5391c0-b2da-4a48-b23d-8d10c28d7f5f" containerName="cleanup" Oct 30 14:30:34.469599 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.469596 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20f347e-1957-4233-9a5a-7e8a1e40410d" containerName="revoke-keys" Oct 30 14:30:34.472854 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.472838 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:34.486138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.486119 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:30:34.486406 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.486385 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:30:34.486406 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.486400 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:30:34.486525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.486421 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:30:34.497189 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.497163 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d"] Oct 30 14:30:34.519738 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.519698 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b442a1b-7616-4e5b-ac26-e5e60723e882" path="/var/lib/kubelet/pods/2b442a1b-7616-4e5b-ac26-e5e60723e882/volumes" Oct 30 14:30:34.520190 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.520174 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96e3e13-1d90-4ab9-8539-70730cb8a3a3" path="/var/lib/kubelet/pods/f96e3e13-1d90-4ab9-8539-70730cb8a3a3/volumes" Oct 30 14:30:34.532335 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.532304 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj98f\" (UniqueName: \"kubernetes.io/projected/cbf43098-a448-435a-a798-c156f5c5cde9-kube-api-access-jj98f\") pod \"maas-api-revoke-sub-d3fba056-tg87d\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:34.532447 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.532380 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tg87d\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:34.532528 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.532474 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tg87d\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:34.633124 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.633084 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tg87d\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:34.633323 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.633187 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj98f\" (UniqueName: \"kubernetes.io/projected/cbf43098-a448-435a-a798-c156f5c5cde9-kube-api-access-jj98f\") pod \"maas-api-revoke-sub-d3fba056-tg87d\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:34.633323 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.633218 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tg87d\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:34.633859 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.633833 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tg87d\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:34.635584 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.635562 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tg87d\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:34.655143 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.655115 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj98f\" (UniqueName: \"kubernetes.io/projected/cbf43098-a448-435a-a798-c156f5c5cde9-kube-api-access-jj98f\") pod \"maas-api-revoke-sub-d3fba056-tg87d\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:34.789475 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:34.789395 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:35.094580 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:35.094547 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d"] Oct 30 14:30:35.096870 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:30:35.096842 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf43098_a448_435a_a798_c156f5c5cde9.slice/crio-aefc40c3429c0fa122582cdbfe5627f9dab20833af9b35acaacfe0cbd099845c WatchSource:0}: Error finding container aefc40c3429c0fa122582cdbfe5627f9dab20833af9b35acaacfe0cbd099845c: Status 404 returned error can't find the container with id aefc40c3429c0fa122582cdbfe5627f9dab20833af9b35acaacfe0cbd099845c Oct 30 14:30:35.206604 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:35.206571 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" event={"ID":"cbf43098-a448-435a-a798-c156f5c5cde9","Type":"ContainerStarted","Data":"aefc40c3429c0fa122582cdbfe5627f9dab20833af9b35acaacfe0cbd099845c"} Oct 30 14:30:35.378147 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:35.378065 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:30:36.218178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:36.218139 4968 generic.go:356] "Generic (PLEG): container finished" podID="cbf43098-a448-435a-a798-c156f5c5cde9" containerID="d59124fe4e910f313d0cc437fa166a64525e5b74d962b43048558e18c6dd29fc" exitCode=0 Oct 30 14:30:36.218605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:36.218204 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" event={"ID":"cbf43098-a448-435a-a798-c156f5c5cde9","Type":"ContainerDied","Data":"d59124fe4e910f313d0cc437fa166a64525e5b74d962b43048558e18c6dd29fc"} Oct 30 14:30:37.665758 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:37.659846 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:37.666216 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:37.666009 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-xks-ca\") pod \"cbf43098-a448-435a-a798-c156f5c5cde9\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " Oct 30 14:30:37.666216 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:37.666064 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-openshift-ca\") pod \"cbf43098-a448-435a-a798-c156f5c5cde9\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " Oct 30 14:30:37.666216 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:37.666147 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf43098-a448-435a-a798-c156f5c5cde9-kube-api-access-jj98f\" (UniqueName: \"kubernetes.io/projected/cbf43098-a448-435a-a798-c156f5c5cde9-kube-api-access-jj98f\") pod \"cbf43098-a448-435a-a798-c156f5c5cde9\" (UID: \"cbf43098-a448-435a-a798-c156f5c5cde9\") " Oct 30 14:30:37.667155 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:37.667102 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-openshift-ca" pod "cbf43098-a448-435a-a798-c156f5c5cde9" (UID: "cbf43098-a448-435a-a798-c156f5c5cde9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:37.669174 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:37.669149 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf43098-a448-435a-a798-c156f5c5cde9-kube-api-access-jj98f" pod "cbf43098-a448-435a-a798-c156f5c5cde9" (UID: "cbf43098-a448-435a-a798-c156f5c5cde9"). InnerVolumeSpecName "kube-api-access-jj98f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:37.669456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:37.669430 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-xks-ca" pod "cbf43098-a448-435a-a798-c156f5c5cde9" (UID: "cbf43098-a448-435a-a798-c156f5c5cde9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:30:37.782757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:37.775395 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj98f\" (UniqueName: \"kubernetes.io/projected/cbf43098-a448-435a-a798-c156f5c5cde9-kube-api-access-jj98f\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:30:37.783013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:37.782989 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:30:37.783133 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:37.783119 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf43098-a448-435a-a798-c156f5c5cde9-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:30:38.263766 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:38.263735 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" Oct 30 14:30:38.264036 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:38.263994 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d" event={"ID":"cbf43098-a448-435a-a798-c156f5c5cde9","Type":"ContainerDied","Data":"aefc40c3429c0fa122582cdbfe5627f9dab20833af9b35acaacfe0cbd099845c"} Oct 30 14:30:38.264192 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:38.264065 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aefc40c3429c0fa122582cdbfe5627f9dab20833af9b35acaacfe0cbd099845c" Oct 30 14:30:45.034413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:45.034369 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp"] Oct 30 14:30:45.043813 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:45.043784 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-399a47bf-b2fdp"] Oct 30 14:30:45.348809 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:45.348778 4968 scope.go:122] "RemoveContainer" containerID="7d04954eb49a9a38d013442930db7929e5f909722aa989ff301dd5effe74ef14" Oct 30 14:30:45.358662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:45.358639 4968 scope.go:122] "RemoveContainer" containerID="1257645e89bf8f5d369f50d2e1bd22472d3939b7b10ffbfda41197ea8a6cef08" Oct 30 14:30:45.371635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:45.371616 4968 scope.go:122] "RemoveContainer" containerID="1b488bcb9d4f6899569f354a16ad616bed9d6233d5237e6a360c30f04005aa28" Oct 30 14:30:45.381533 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:45.381511 4968 scope.go:122] "RemoveContainer" containerID="8712ac0cccc2f922178f90ac51906db55ec6b1dac2f5715798d72b7cfbb5e6b7" Oct 30 14:30:45.586012 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:45.585965 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:30:46.517759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:46.517710 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd8c616-3730-4f1e-b0e5-614c7f407612" path="/var/lib/kubelet/pods/9dd8c616-3730-4f1e-b0e5-614c7f407612/volumes" Oct 30 14:30:50.275397 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:50.275362 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:30:50.444459 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:30:50.444425 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:30:56.869103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:56.869070 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2"] Oct 30 14:30:56.869531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:56.869492 4968 cpu_manager.go:414] "containerMap: removing container" podUID="cbf43098-a448-435a-a798-c156f5c5cde9" containerName="revoke-keys" Oct 30 14:30:56.869531 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:56.869503 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf43098-a448-435a-a798-c156f5c5cde9" containerName="revoke-keys" Oct 30 14:30:56.869617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:56.869588 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf43098-a448-435a-a798-c156f5c5cde9" containerName="revoke-keys" Oct 30 14:30:56.874678 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:56.874653 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:56.893163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:56.893138 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2"] Oct 30 14:30:56.900010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:56.899974 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:30:56.900010 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:56.900004 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:30:56.900224 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:56.900004 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:30:56.900224 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:56.900096 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:30:57.002699 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.002671 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gbws2\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:57.002877 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.002782 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gbws2\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:57.002877 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.002846 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ct8\" (UniqueName: \"kubernetes.io/projected/c8123d50-7fb9-4d4e-993c-01d4dc12a002-kube-api-access-f8ct8\") pod \"maas-api-revoke-sub-ae594979-gbws2\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:57.035665 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.035634 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk"] Oct 30 14:30:57.038483 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.038458 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb063a2-ntpjk"] Oct 30 14:30:57.103963 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.103932 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gbws2\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:57.104114 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.104001 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gbws2\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:57.104114 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.104028 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ct8\" (UniqueName: \"kubernetes.io/projected/c8123d50-7fb9-4d4e-993c-01d4dc12a002-kube-api-access-f8ct8\") pod \"maas-api-revoke-sub-ae594979-gbws2\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:57.104563 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.104544 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gbws2\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:57.106319 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.106294 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gbws2\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:57.125757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.125690 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ct8\" (UniqueName: \"kubernetes.io/projected/c8123d50-7fb9-4d4e-993c-01d4dc12a002-kube-api-access-f8ct8\") pod \"maas-api-revoke-sub-ae594979-gbws2\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:57.187575 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.187548 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:57.488139 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:57.488115 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2"] Oct 30 14:30:57.490859 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:30:57.490827 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8123d50_7fb9_4d4e_993c_01d4dc12a002.slice/crio-f39dfe5474bc7cf16e3ad7d4b7a449d7e00c23b262825e2701b44afa9d204022 WatchSource:0}: Error finding container f39dfe5474bc7cf16e3ad7d4b7a449d7e00c23b262825e2701b44afa9d204022: Status 404 returned error can't find the container with id f39dfe5474bc7cf16e3ad7d4b7a449d7e00c23b262825e2701b44afa9d204022 Oct 30 14:30:58.405506 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:58.405474 4968 generic.go:356] "Generic (PLEG): container finished" podID="c8123d50-7fb9-4d4e-993c-01d4dc12a002" containerID="287d5d675f64bf5bb5d86569db7d7c5aa71c037a2d614d63409806f3657d542a" exitCode=0 Oct 30 14:30:58.405923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:58.405559 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" event={"ID":"c8123d50-7fb9-4d4e-993c-01d4dc12a002","Type":"ContainerDied","Data":"287d5d675f64bf5bb5d86569db7d7c5aa71c037a2d614d63409806f3657d542a"} Oct 30 14:30:58.405923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:58.405626 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" event={"ID":"c8123d50-7fb9-4d4e-993c-01d4dc12a002","Type":"ContainerStarted","Data":"f39dfe5474bc7cf16e3ad7d4b7a449d7e00c23b262825e2701b44afa9d204022"} Oct 30 14:30:58.521333 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:58.521304 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb89876-ad55-431b-b338-e8b22313c4ab" path="/var/lib/kubelet/pods/efb89876-ad55-431b-b338-e8b22313c4ab/volumes" Oct 30 14:30:59.030709 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.030675 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52"] Oct 30 14:30:59.033353 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.033327 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47849a1e-m6g52"] Oct 30 14:30:59.556634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.556611 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:30:59.626741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.626681 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-xks-ca\") pod \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " Oct 30 14:30:59.626901 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.626751 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-openshift-ca\") pod \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " Oct 30 14:30:59.626975 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.626901 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8123d50-7fb9-4d4e-993c-01d4dc12a002-kube-api-access-f8ct8\" (UniqueName: \"kubernetes.io/projected/c8123d50-7fb9-4d4e-993c-01d4dc12a002-kube-api-access-f8ct8\") pod \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\" (UID: \"c8123d50-7fb9-4d4e-993c-01d4dc12a002\") " Oct 30 14:30:59.627076 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.627054 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-openshift-ca" pod "c8123d50-7fb9-4d4e-993c-01d4dc12a002" (UID: "c8123d50-7fb9-4d4e-993c-01d4dc12a002"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:59.627243 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.627229 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:30:59.628702 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.628676 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-xks-ca" pod "c8123d50-7fb9-4d4e-993c-01d4dc12a002" (UID: "c8123d50-7fb9-4d4e-993c-01d4dc12a002"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:30:59.628837 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.628821 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8123d50-7fb9-4d4e-993c-01d4dc12a002-kube-api-access-f8ct8" pod "c8123d50-7fb9-4d4e-993c-01d4dc12a002" (UID: "c8123d50-7fb9-4d4e-993c-01d4dc12a002"). InnerVolumeSpecName "kube-api-access-f8ct8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:59.728598 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.728517 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8123d50-7fb9-4d4e-993c-01d4dc12a002-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:30:59.728829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:30:59.728812 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8ct8\" (UniqueName: \"kubernetes.io/projected/c8123d50-7fb9-4d4e-993c-01d4dc12a002-kube-api-access-f8ct8\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:31:00.419536 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:00.419505 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" Oct 30 14:31:00.419740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:00.419535 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2" event={"ID":"c8123d50-7fb9-4d4e-993c-01d4dc12a002","Type":"ContainerDied","Data":"f39dfe5474bc7cf16e3ad7d4b7a449d7e00c23b262825e2701b44afa9d204022"} Oct 30 14:31:00.419740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:00.419622 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f39dfe5474bc7cf16e3ad7d4b7a449d7e00c23b262825e2701b44afa9d204022" Oct 30 14:31:00.517207 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:00.517174 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41e256b-af47-49bb-a8a0-a9374935fab8" path="/var/lib/kubelet/pods/e41e256b-af47-49bb-a8a0-a9374935fab8/volumes" Oct 30 14:31:02.032233 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:02.032200 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r"] Oct 30 14:31:02.040134 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:02.040107 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9dcb4e-qcc8r"] Oct 30 14:31:02.517644 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:02.517612 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b66c2f-7a81-4340-a5ed-c539605c2389" path="/var/lib/kubelet/pods/e2b66c2f-7a81-4340-a5ed-c539605c2389/volumes" Oct 30 14:31:04.030606 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:04.030574 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj"] Oct 30 14:31:04.040441 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:04.040409 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk"] Oct 30 14:31:04.043281 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:04.043255 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd21d625-b9kdj"] Oct 30 14:31:04.045541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:04.045520 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e4c39a-brnvk"] Oct 30 14:31:04.517349 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:04.517317 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9c05ff-d1c5-4e2a-80b7-143b4757e660" path="/var/lib/kubelet/pods/2b9c05ff-d1c5-4e2a-80b7-143b4757e660/volumes" Oct 30 14:31:04.517748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:04.517711 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d34ecd5-3570-487d-b86c-d315737475d5" path="/var/lib/kubelet/pods/9d34ecd5-3570-487d-b86c-d315737475d5/volumes" Oct 30 14:31:08.135333 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:08.135284 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:31:09.031586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:09.031549 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9"] Oct 30 14:31:09.040909 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:09.040883 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e945de7-q9sd9"] Oct 30 14:31:10.030042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:10.030009 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck"] Oct 30 14:31:10.039574 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:10.039550 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205590bb-txwck"] Oct 30 14:31:10.517556 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:10.517514 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2732b5e3-df26-4e6c-927b-72edc4859022" path="/var/lib/kubelet/pods/2732b5e3-df26-4e6c-927b-72edc4859022/volumes" Oct 30 14:31:10.518074 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:10.518050 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcbbc581-0989-460d-b26d-2dc65b74974e" path="/var/lib/kubelet/pods/bcbbc581-0989-460d-b26d-2dc65b74974e/volumes" Oct 30 14:31:13.258545 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:13.258504 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:31:18.419691 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.419657 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2"] Oct 30 14:31:18.420158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.420144 4968 cpu_manager.go:414] "containerMap: removing container" podUID="c8123d50-7fb9-4d4e-993c-01d4dc12a002" containerName="revoke-keys" Oct 30 14:31:18.420203 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.420160 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8123d50-7fb9-4d4e-993c-01d4dc12a002" containerName="revoke-keys" Oct 30 14:31:18.420266 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.420256 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8123d50-7fb9-4d4e-993c-01d4dc12a002" containerName="revoke-keys" Oct 30 14:31:18.424873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.424856 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:18.444639 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.444616 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2"] Oct 30 14:31:18.450970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.450938 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:31:18.451233 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.451211 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:31:18.451233 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.451211 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:31:18.451417 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.451232 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:31:18.519816 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.519784 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgl5k\" (UniqueName: \"kubernetes.io/projected/b9648934-606c-47ba-b8d2-fde42fbfb37b-kube-api-access-vgl5k\") pod \"maas-api-revoke-sub-a4aef7d7-fzdv2\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:18.519996 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.519852 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzdv2\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:18.519996 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.519941 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzdv2\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:18.620606 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.620570 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzdv2\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:18.620606 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.620613 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzdv2\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:18.620868 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.620680 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgl5k\" (UniqueName: \"kubernetes.io/projected/b9648934-606c-47ba-b8d2-fde42fbfb37b-kube-api-access-vgl5k\") pod \"maas-api-revoke-sub-a4aef7d7-fzdv2\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:18.621332 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.621305 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzdv2\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:18.623032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.623009 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzdv2\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:18.643261 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.643231 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgl5k\" (UniqueName: \"kubernetes.io/projected/b9648934-606c-47ba-b8d2-fde42fbfb37b-kube-api-access-vgl5k\") pod \"maas-api-revoke-sub-a4aef7d7-fzdv2\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:18.737823 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:18.737708 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:19.039127 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:19.039100 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2"] Oct 30 14:31:19.042026 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:31:19.041991 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9648934_606c_47ba_b8d2_fde42fbfb37b.slice/crio-4c9b4279163b77f5a5deb8cf92af8cd4d05626d3602f6d28136b3c928f134e62 WatchSource:0}: Error finding container 4c9b4279163b77f5a5deb8cf92af8cd4d05626d3602f6d28136b3c928f134e62: Status 404 returned error can't find the container with id 4c9b4279163b77f5a5deb8cf92af8cd4d05626d3602f6d28136b3c928f134e62 Oct 30 14:31:19.587963 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:19.587926 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" event={"ID":"b9648934-606c-47ba-b8d2-fde42fbfb37b","Type":"ContainerStarted","Data":"f35b9553c8d1ce0a71458d735b880223b213f54bf7c82a376de2517a21b7d3dd"} Oct 30 14:31:19.588336 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:19.587981 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" event={"ID":"b9648934-606c-47ba-b8d2-fde42fbfb37b","Type":"ContainerStarted","Data":"4c9b4279163b77f5a5deb8cf92af8cd4d05626d3602f6d28136b3c928f134e62"} Oct 30 14:31:19.625438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:19.625371 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" podStartSLOduration=1.6253580520000002 podStartE2EDuration="1.625358052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:19.624822438 +0000 UTC m=+1475.250265811" watchObservedRunningTime="2026-10-30 14:31:19.625358052 +0000 UTC m=+1475.250801475" Oct 30 14:31:20.443179 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:31:20.443141 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:31:20.596682 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:20.596651 4968 generic.go:356] "Generic (PLEG): container finished" podID="b9648934-606c-47ba-b8d2-fde42fbfb37b" containerID="f35b9553c8d1ce0a71458d735b880223b213f54bf7c82a376de2517a21b7d3dd" exitCode=0 Oct 30 14:31:20.597122 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:20.596688 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" event={"ID":"b9648934-606c-47ba-b8d2-fde42fbfb37b","Type":"ContainerDied","Data":"f35b9553c8d1ce0a71458d735b880223b213f54bf7c82a376de2517a21b7d3dd"} Oct 30 14:31:21.764923 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:21.764892 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:21.862451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:21.862374 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9648934-606c-47ba-b8d2-fde42fbfb37b-kube-api-access-vgl5k\" (UniqueName: \"kubernetes.io/projected/b9648934-606c-47ba-b8d2-fde42fbfb37b-kube-api-access-vgl5k\") pod \"b9648934-606c-47ba-b8d2-fde42fbfb37b\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " Oct 30 14:31:21.862451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:21.862419 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-xks-ca\") pod \"b9648934-606c-47ba-b8d2-fde42fbfb37b\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " Oct 30 14:31:21.862638 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:21.862516 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-openshift-ca\") pod \"b9648934-606c-47ba-b8d2-fde42fbfb37b\" (UID: \"b9648934-606c-47ba-b8d2-fde42fbfb37b\") " Oct 30 14:31:21.862929 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:21.862903 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-openshift-ca" pod "b9648934-606c-47ba-b8d2-fde42fbfb37b" (UID: "b9648934-606c-47ba-b8d2-fde42fbfb37b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:21.864498 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:21.864473 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-xks-ca" pod "b9648934-606c-47ba-b8d2-fde42fbfb37b" (UID: "b9648934-606c-47ba-b8d2-fde42fbfb37b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:21.864599 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:21.864536 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9648934-606c-47ba-b8d2-fde42fbfb37b-kube-api-access-vgl5k" pod "b9648934-606c-47ba-b8d2-fde42fbfb37b" (UID: "b9648934-606c-47ba-b8d2-fde42fbfb37b"). InnerVolumeSpecName "kube-api-access-vgl5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:21.963339 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:21.963306 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:31:21.963339 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:21.963336 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgl5k\" (UniqueName: \"kubernetes.io/projected/b9648934-606c-47ba-b8d2-fde42fbfb37b-kube-api-access-vgl5k\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:31:21.963339 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:21.963346 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9648934-606c-47ba-b8d2-fde42fbfb37b-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.619882 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:22.619853 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" Oct 30 14:31:22.620057 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:22.619884 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2" event={"ID":"b9648934-606c-47ba-b8d2-fde42fbfb37b","Type":"ContainerDied","Data":"4c9b4279163b77f5a5deb8cf92af8cd4d05626d3602f6d28136b3c928f134e62"} Oct 30 14:31:22.620057 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:22.619936 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c9b4279163b77f5a5deb8cf92af8cd4d05626d3602f6d28136b3c928f134e62" Oct 30 14:31:29.667602 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:29.667525 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:31:34.070830 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:34.070793 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:31:36.650768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:36.649903 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:31:36.650768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:36.649964 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z2bwp" podUID="43dc1f23-75ac-490a-9594-6ba78cb3d01e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:31:36.742739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:36.741256 4968 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:31:36.742739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:36.741325 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zv6j4" podUID="ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:31:38.483334 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.483297 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5"] Oct 30 14:31:38.483822 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.483805 4968 cpu_manager.go:414] "containerMap: removing container" podUID="b9648934-606c-47ba-b8d2-fde42fbfb37b" containerName="revoke-keys" Oct 30 14:31:38.483891 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.483824 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9648934-606c-47ba-b8d2-fde42fbfb37b" containerName="revoke-keys" Oct 30 14:31:38.483934 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.483924 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9648934-606c-47ba-b8d2-fde42fbfb37b" containerName="revoke-keys" Oct 30 14:31:38.487327 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.487311 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:38.500653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.500626 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:31:38.510484 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.510462 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:31:38.510616 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.510532 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:31:38.510706 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.510689 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:31:38.521190 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.521158 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5"] Oct 30 14:31:38.571185 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.571152 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fzlq5\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:38.571364 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.571192 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fzlq5\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:38.571364 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.571238 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4sq\" (UniqueName: \"kubernetes.io/projected/00d95909-1ae3-4b27-a135-c63b9a80d912-kube-api-access-gb4sq\") pod \"maas-api-revoke-sub-79805ca9-fzlq5\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:38.671951 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.671919 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4sq\" (UniqueName: \"kubernetes.io/projected/00d95909-1ae3-4b27-a135-c63b9a80d912-kube-api-access-gb4sq\") pod \"maas-api-revoke-sub-79805ca9-fzlq5\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:38.672150 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.672061 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fzlq5\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:38.672150 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.672093 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fzlq5\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:38.672826 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.672797 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fzlq5\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:38.674509 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.674486 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fzlq5\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:38.694272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.694238 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4sq\" (UniqueName: \"kubernetes.io/projected/00d95909-1ae3-4b27-a135-c63b9a80d912-kube-api-access-gb4sq\") pod \"maas-api-revoke-sub-79805ca9-fzlq5\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:38.802414 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:38.802320 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:39.103739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:39.103698 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5"] Oct 30 14:31:39.106455 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:31:39.106424 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d95909_1ae3_4b27_a135_c63b9a80d912.slice/crio-5f2b508c05dad12266e2971801faa98c804d84a9e5f333052477ac585ebfc328 WatchSource:0}: Error finding container 5f2b508c05dad12266e2971801faa98c804d84a9e5f333052477ac585ebfc328: Status 404 returned error can't find the container with id 5f2b508c05dad12266e2971801faa98c804d84a9e5f333052477ac585ebfc328 Oct 30 14:31:39.780058 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:39.779972 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" event={"ID":"00d95909-1ae3-4b27-a135-c63b9a80d912","Type":"ContainerStarted","Data":"45bceb38635bd0a9f4d6aa5c713a3b9424b8892ad665b367c025890320d83330"} Oct 30 14:31:39.780058 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:39.780031 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" event={"ID":"00d95909-1ae3-4b27-a135-c63b9a80d912","Type":"ContainerStarted","Data":"5f2b508c05dad12266e2971801faa98c804d84a9e5f333052477ac585ebfc328"} Oct 30 14:31:39.819797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:39.819743 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" podStartSLOduration=1.819727114 podStartE2EDuration="1.819727114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:39.817887279 +0000 UTC m=+1495.443330653" watchObservedRunningTime="2026-10-30 14:31:39.819727114 +0000 UTC m=+1495.445170478" Oct 30 14:31:40.788662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:40.788631 4968 generic.go:356] "Generic (PLEG): container finished" podID="00d95909-1ae3-4b27-a135-c63b9a80d912" containerID="45bceb38635bd0a9f4d6aa5c713a3b9424b8892ad665b367c025890320d83330" exitCode=0 Oct 30 14:31:40.789111 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:40.788737 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" event={"ID":"00d95909-1ae3-4b27-a135-c63b9a80d912","Type":"ContainerDied","Data":"45bceb38635bd0a9f4d6aa5c713a3b9424b8892ad665b367c025890320d83330"} Oct 30 14:31:41.935502 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:41.935476 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:42.007510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.007470 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d95909-1ae3-4b27-a135-c63b9a80d912-kube-api-access-gb4sq\" (UniqueName: \"kubernetes.io/projected/00d95909-1ae3-4b27-a135-c63b9a80d912-kube-api-access-gb4sq\") pod \"00d95909-1ae3-4b27-a135-c63b9a80d912\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " Oct 30 14:31:42.007757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.007594 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-openshift-ca\") pod \"00d95909-1ae3-4b27-a135-c63b9a80d912\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " Oct 30 14:31:42.007757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.007697 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-xks-ca\") pod \"00d95909-1ae3-4b27-a135-c63b9a80d912\" (UID: \"00d95909-1ae3-4b27-a135-c63b9a80d912\") " Oct 30 14:31:42.008020 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.007987 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-openshift-ca" pod "00d95909-1ae3-4b27-a135-c63b9a80d912" (UID: "00d95909-1ae3-4b27-a135-c63b9a80d912"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:42.008152 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.008104 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:31:42.010057 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.010026 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-xks-ca" pod "00d95909-1ae3-4b27-a135-c63b9a80d912" (UID: "00d95909-1ae3-4b27-a135-c63b9a80d912"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:42.010245 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.010222 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d95909-1ae3-4b27-a135-c63b9a80d912-kube-api-access-gb4sq" pod "00d95909-1ae3-4b27-a135-c63b9a80d912" (UID: "00d95909-1ae3-4b27-a135-c63b9a80d912"). InnerVolumeSpecName "kube-api-access-gb4sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:42.109437 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.109407 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d95909-1ae3-4b27-a135-c63b9a80d912-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:31:42.109437 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.109434 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4sq\" (UniqueName: \"kubernetes.io/projected/00d95909-1ae3-4b27-a135-c63b9a80d912-kube-api-access-gb4sq\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:31:42.803025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.802942 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" Oct 30 14:31:42.803025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.802976 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5" event={"ID":"00d95909-1ae3-4b27-a135-c63b9a80d912","Type":"ContainerDied","Data":"5f2b508c05dad12266e2971801faa98c804d84a9e5f333052477ac585ebfc328"} Oct 30 14:31:42.803212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:42.803026 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2b508c05dad12266e2971801faa98c804d84a9e5f333052477ac585ebfc328" Oct 30 14:31:44.708905 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:44.708878 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:31:44.713220 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:44.713200 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:31:45.440067 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.440032 4968 scope.go:122] "RemoveContainer" containerID="996cdb93718435f9fd3c72a95148eca08c4c8a51625e605a3537864a5c88d432" Oct 30 14:31:45.451390 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.451362 4968 scope.go:122] "RemoveContainer" containerID="6da50b2cd572072418f95ad6caf89e591f2f1a3ab6d5ffa7508934e8a06a73c5" Oct 30 14:31:45.460917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.460898 4968 scope.go:122] "RemoveContainer" containerID="8cb07c92c9f1604157a125f3a327dbf44691e0b171cfdc3a71206c768d9a7d37" Oct 30 14:31:45.469084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.469065 4968 scope.go:122] "RemoveContainer" containerID="5a3d59e8e44923b181cf41fa600ba6e8b8ebb2019fc0747e27a5c494c0eb46b7" Oct 30 14:31:45.476908 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.476892 4968 scope.go:122] "RemoveContainer" containerID="219d2978550e97ff6c5a79639cc1dc544717889d85710e8a1c98d183efff983b" Oct 30 14:31:45.485153 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.485134 4968 scope.go:122] "RemoveContainer" containerID="df54c2d9c97256d825e609485084482e3ebaf0f7a6393097c38f713ff2909fcc" Oct 30 14:31:45.493103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.493081 4968 scope.go:122] "RemoveContainer" containerID="8b0ab3e410d90b68937ebeee35b2e09d927b6c859960c8ff053d08b5e2e45d38" Oct 30 14:31:45.501101 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.501075 4968 scope.go:122] "RemoveContainer" containerID="bb8052c22c1d9e6baca2473b8d40421f243e199337a14c74eacf0cf8336320db" Oct 30 14:31:45.509087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.509071 4968 scope.go:122] "RemoveContainer" containerID="8dba7b8e855cc93aca1521cb66a61128774a50a976f41d15a51c92ebc83ddf4b" Oct 30 14:31:45.516815 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.516792 4968 scope.go:122] "RemoveContainer" containerID="8e0f86e8fb8214489906e27ae81687b25ab86d1d0177a93b04f825cca7a8b547" Oct 30 14:31:45.524705 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.524687 4968 scope.go:122] "RemoveContainer" containerID="1443a6a7f0f1a81fec5ff2eece37dc3049ad50c7d6c5523acafeac2b1add46bc" Oct 30 14:31:45.532706 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.532688 4968 scope.go:122] "RemoveContainer" containerID="0c121ad35f675fd7403e14db1fa590b770ad008de6db1435a1a99e428e49efc0" Oct 30 14:31:45.540545 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.540524 4968 scope.go:122] "RemoveContainer" containerID="d5434bc562471e976455f800d082ab02af6975daacadf017d77b8fbc11a4dfb7" Oct 30 14:31:45.548615 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.548597 4968 scope.go:122] "RemoveContainer" containerID="ec9242e4a5b07ce7db8bc622508448d8817644548a0699f2c495f6068be89904" Oct 30 14:31:45.556848 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:45.556831 4968 scope.go:122] "RemoveContainer" containerID="9dd3e7175c8146b0effe3445f5a2fdae65fff0bc3b11fd7a8a99fd58cf9af47b" Oct 30 14:31:49.667191 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:49.667157 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:31:50.443784 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:31:50.443749 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:31:54.770396 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:54.770358 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:31:59.032527 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.032489 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt"] Oct 30 14:31:59.042074 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.042041 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cb2b3e-5jzgt"] Oct 30 14:31:59.504940 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.504907 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9"] Oct 30 14:31:59.505340 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.505327 4968 cpu_manager.go:414] "containerMap: removing container" podUID="00d95909-1ae3-4b27-a135-c63b9a80d912" containerName="revoke-keys" Oct 30 14:31:59.505388 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.505341 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d95909-1ae3-4b27-a135-c63b9a80d912" containerName="revoke-keys" Oct 30 14:31:59.505445 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.505436 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d95909-1ae3-4b27-a135-c63b9a80d912" containerName="revoke-keys" Oct 30 14:31:59.510512 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.510494 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:31:59.528275 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.528222 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9"] Oct 30 14:31:59.535435 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.535410 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:31:59.535601 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.535412 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:31:59.535601 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.535451 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:31:59.535808 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.535477 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:31:59.618399 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.618366 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bpdj9\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:31:59.618568 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.618413 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncp4h\" (UniqueName: \"kubernetes.io/projected/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-kube-api-access-ncp4h\") pod \"maas-api-revoke-sub-0b7e8edb-bpdj9\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:31:59.618568 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.618526 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bpdj9\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:31:59.719664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.719632 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bpdj9\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:31:59.719886 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.719692 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bpdj9\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:31:59.719886 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.719763 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncp4h\" (UniqueName: \"kubernetes.io/projected/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-kube-api-access-ncp4h\") pod \"maas-api-revoke-sub-0b7e8edb-bpdj9\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:31:59.720312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.720286 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bpdj9\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:31:59.721994 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.721968 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bpdj9\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:31:59.741862 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.741838 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncp4h\" (UniqueName: \"kubernetes.io/projected/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-kube-api-access-ncp4h\") pod \"maas-api-revoke-sub-0b7e8edb-bpdj9\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:31:59.822942 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:31:59.822871 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:32:00.124978 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:00.124952 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9"] Oct 30 14:32:00.127516 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:32:00.127487 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f45bc9_ac20_449f_a077_99cb1f8fb6c7.slice/crio-62b38d6eb09bcb806e62ae99edd3c81ee03f57a1bf2b5979850155ab67b47753 WatchSource:0}: Error finding container 62b38d6eb09bcb806e62ae99edd3c81ee03f57a1bf2b5979850155ab67b47753: Status 404 returned error can't find the container with id 62b38d6eb09bcb806e62ae99edd3c81ee03f57a1bf2b5979850155ab67b47753 Oct 30 14:32:00.518213 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:00.518179 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2337deb7-cb13-44d4-b3f4-309eb5c7c99d" path="/var/lib/kubelet/pods/2337deb7-cb13-44d4-b3f4-309eb5c7c99d/volumes" Oct 30 14:32:00.927922 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:00.927893 4968 generic.go:356] "Generic (PLEG): container finished" podID="a7f45bc9-ac20-449f-a077-99cb1f8fb6c7" containerID="fcdd31a58982c85d73b9f656bd7cbba3b2eaa9c3e12917bb736783a9b0e1271c" exitCode=0 Oct 30 14:32:00.928084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:00.927980 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" event={"ID":"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7","Type":"ContainerDied","Data":"fcdd31a58982c85d73b9f656bd7cbba3b2eaa9c3e12917bb736783a9b0e1271c"} Oct 30 14:32:00.928084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:00.928042 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" event={"ID":"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7","Type":"ContainerStarted","Data":"62b38d6eb09bcb806e62ae99edd3c81ee03f57a1bf2b5979850155ab67b47753"} Oct 30 14:32:02.075775 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.075751 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:32:02.143078 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.143037 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-xks-ca\") pod \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " Oct 30 14:32:02.143253 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.143086 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-kube-api-access-ncp4h\" (UniqueName: \"kubernetes.io/projected/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-kube-api-access-ncp4h\") pod \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " Oct 30 14:32:02.143253 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.143132 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-openshift-ca\") pod \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\" (UID: \"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7\") " Oct 30 14:32:02.143615 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.143586 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-openshift-ca" pod "a7f45bc9-ac20-449f-a077-99cb1f8fb6c7" (UID: "a7f45bc9-ac20-449f-a077-99cb1f8fb6c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:02.145198 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.145166 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-xks-ca" pod "a7f45bc9-ac20-449f-a077-99cb1f8fb6c7" (UID: "a7f45bc9-ac20-449f-a077-99cb1f8fb6c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:32:02.145322 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.145227 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-kube-api-access-ncp4h" pod "a7f45bc9-ac20-449f-a077-99cb1f8fb6c7" (UID: "a7f45bc9-ac20-449f-a077-99cb1f8fb6c7"). InnerVolumeSpecName "kube-api-access-ncp4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:02.244694 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.244605 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:02.244694 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.244639 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:02.244694 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.244649 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncp4h\" (UniqueName: \"kubernetes.io/projected/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7-kube-api-access-ncp4h\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:02.942113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.942085 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" Oct 30 14:32:02.942290 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.942079 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9" event={"ID":"a7f45bc9-ac20-449f-a077-99cb1f8fb6c7","Type":"ContainerDied","Data":"62b38d6eb09bcb806e62ae99edd3c81ee03f57a1bf2b5979850155ab67b47753"} Oct 30 14:32:02.942290 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:02.942214 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62b38d6eb09bcb806e62ae99edd3c81ee03f57a1bf2b5979850155ab67b47753" Oct 30 14:32:10.213441 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.213404 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd"] Oct 30 14:32:10.213954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.213868 4968 cpu_manager.go:414] "containerMap: removing container" podUID="a7f45bc9-ac20-449f-a077-99cb1f8fb6c7" containerName="revoke-keys" Oct 30 14:32:10.213954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.213880 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f45bc9-ac20-449f-a077-99cb1f8fb6c7" containerName="revoke-keys" Oct 30 14:32:10.213954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.213948 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f45bc9-ac20-449f-a077-99cb1f8fb6c7" containerName="revoke-keys" Oct 30 14:32:10.217071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.217050 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:10.230252 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.230226 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:32:10.239748 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.239678 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:32:10.240001 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.239968 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:32:10.240150 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.240088 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:32:10.244373 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.244349 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd"] Oct 30 14:32:10.319935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.319900 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdg28\" (UniqueName: \"kubernetes.io/projected/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-kube-api-access-wdg28\") pod \"maas-api-revoke-sub-3d4156f1-kk7zd\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:10.319935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.319945 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kk7zd\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:10.320143 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.319983 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kk7zd\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:10.420850 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.420820 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdg28\" (UniqueName: \"kubernetes.io/projected/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-kube-api-access-wdg28\") pod \"maas-api-revoke-sub-3d4156f1-kk7zd\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:10.421041 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.420860 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kk7zd\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:10.421041 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.420915 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kk7zd\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:10.421581 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.421556 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kk7zd\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:10.423422 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.423394 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kk7zd\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:10.443387 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.443363 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdg28\" (UniqueName: \"kubernetes.io/projected/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-kube-api-access-wdg28\") pod \"maas-api-revoke-sub-3d4156f1-kk7zd\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:10.531418 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.531345 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:10.683485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.683449 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:32:10.840317 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.840293 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd"] Oct 30 14:32:10.843080 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:32:10.843048 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f2aa45_0a1b_4ee8_9dc6_119fd1638820.slice/crio-1a94d25427971b9b1c4f38edd678b6f74577f0cc392371496c0806cda4d3072f WatchSource:0}: Error finding container 1a94d25427971b9b1c4f38edd678b6f74577f0cc392371496c0806cda4d3072f: Status 404 returned error can't find the container with id 1a94d25427971b9b1c4f38edd678b6f74577f0cc392371496c0806cda4d3072f Oct 30 14:32:10.993131 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:10.993094 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" event={"ID":"76f2aa45-0a1b-4ee8-9dc6-119fd1638820","Type":"ContainerStarted","Data":"1a94d25427971b9b1c4f38edd678b6f74577f0cc392371496c0806cda4d3072f"} Oct 30 14:32:12.001255 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:12.001161 4968 generic.go:356] "Generic (PLEG): container finished" podID="76f2aa45-0a1b-4ee8-9dc6-119fd1638820" containerID="2dfbc420da616a1a1cb54e4a89c5401b9d17725d074737f4a8f519f523552a5c" exitCode=0 Oct 30 14:32:12.001673 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:12.001252 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" event={"ID":"76f2aa45-0a1b-4ee8-9dc6-119fd1638820","Type":"ContainerDied","Data":"2dfbc420da616a1a1cb54e4a89c5401b9d17725d074737f4a8f519f523552a5c"} Oct 30 14:32:13.148765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:13.148741 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:13.247131 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:13.247100 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-openshift-ca\") pod \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " Oct 30 14:32:13.247303 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:13.247189 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-xks-ca\") pod \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " Oct 30 14:32:13.247303 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:13.247254 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-kube-api-access-wdg28\" (UniqueName: \"kubernetes.io/projected/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-kube-api-access-wdg28\") pod \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\" (UID: \"76f2aa45-0a1b-4ee8-9dc6-119fd1638820\") " Oct 30 14:32:13.247550 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:13.247521 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-openshift-ca" pod "76f2aa45-0a1b-4ee8-9dc6-119fd1638820" (UID: "76f2aa45-0a1b-4ee8-9dc6-119fd1638820"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:13.249368 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:13.249343 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-kube-api-access-wdg28" pod "76f2aa45-0a1b-4ee8-9dc6-119fd1638820" (UID: "76f2aa45-0a1b-4ee8-9dc6-119fd1638820"). InnerVolumeSpecName "kube-api-access-wdg28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:13.249471 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:13.249417 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-xks-ca" pod "76f2aa45-0a1b-4ee8-9dc6-119fd1638820" (UID: "76f2aa45-0a1b-4ee8-9dc6-119fd1638820"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:32:13.348177 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:13.348152 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:13.348177 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:13.348174 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdg28\" (UniqueName: \"kubernetes.io/projected/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-kube-api-access-wdg28\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:13.348330 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:13.348185 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f2aa45-0a1b-4ee8-9dc6-119fd1638820-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:14.016136 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:14.016097 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" Oct 30 14:32:14.016136 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:14.016124 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd" event={"ID":"76f2aa45-0a1b-4ee8-9dc6-119fd1638820","Type":"ContainerDied","Data":"1a94d25427971b9b1c4f38edd678b6f74577f0cc392371496c0806cda4d3072f"} Oct 30 14:32:14.016345 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:14.016205 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a94d25427971b9b1c4f38edd678b6f74577f0cc392371496c0806cda4d3072f" Oct 30 14:32:17.038514 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:17.038488 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr"] Oct 30 14:32:17.047539 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:17.047516 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba7ab5f-247fr"] Oct 30 14:32:18.518100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:18.518068 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35222d09-b538-4534-869f-286bcb99de9d" path="/var/lib/kubelet/pods/35222d09-b538-4534-869f-286bcb99de9d/volumes" Oct 30 14:32:20.444107 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:32:20.444065 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:32:21.448543 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:21.448500 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:32:22.035486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:22.035440 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm"] Oct 30 14:32:22.045222 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:22.045162 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff"] Oct 30 14:32:22.048829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:22.048800 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f48cb7-2gsmm"] Oct 30 14:32:22.050782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:22.050759 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242a80bc-bb8ff"] Oct 30 14:32:22.518580 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:22.518533 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e" path="/var/lib/kubelet/pods/a69e8ee8-c3dd-4028-b3e8-e394d3ce0e9e/volumes" Oct 30 14:32:22.519154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:22.519133 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd61a99c-6d0b-4a82-935a-6fc3d14bacc6" path="/var/lib/kubelet/pods/dd61a99c-6d0b-4a82-935a-6fc3d14bacc6/volumes" Oct 30 14:32:26.739767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:26.739710 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:32:31.456358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.456320 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd"] Oct 30 14:32:31.466042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.466010 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kk7zd"] Oct 30 14:32:31.503358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.503328 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk"] Oct 30 14:32:31.504025 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.504007 4968 cpu_manager.go:414] "containerMap: removing container" podUID="76f2aa45-0a1b-4ee8-9dc6-119fd1638820" containerName="revoke-keys" Oct 30 14:32:31.504075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.504029 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f2aa45-0a1b-4ee8-9dc6-119fd1638820" containerName="revoke-keys" Oct 30 14:32:31.504182 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.504169 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f2aa45-0a1b-4ee8-9dc6-119fd1638820" containerName="revoke-keys" Oct 30 14:32:31.509336 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.509318 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:31.523414 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.523381 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:32:31.523518 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.523415 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:32:31.524032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.524012 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:32:31.526004 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.525981 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk"] Oct 30 14:32:31.532838 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.532819 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:32:31.629937 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.629901 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fn7q\" (UniqueName: \"kubernetes.io/projected/eed7bb55-9c3a-4b64-b2e1-d923d8630240-kube-api-access-4fn7q\") pod \"maas-api-revoke-sub-3d4156f1-mtjkk\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:31.630151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.629961 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mtjkk\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:31.630151 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.630097 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mtjkk\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:31.731380 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.731301 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mtjkk\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:31.731380 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.731369 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fn7q\" (UniqueName: \"kubernetes.io/projected/eed7bb55-9c3a-4b64-b2e1-d923d8630240-kube-api-access-4fn7q\") pod \"maas-api-revoke-sub-3d4156f1-mtjkk\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:31.731634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.731410 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mtjkk\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:31.732134 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.732111 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mtjkk\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:31.733754 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.733733 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mtjkk\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:31.752935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.752909 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fn7q\" (UniqueName: \"kubernetes.io/projected/eed7bb55-9c3a-4b64-b2e1-d923d8630240-kube-api-access-4fn7q\") pod \"maas-api-revoke-sub-3d4156f1-mtjkk\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:31.823678 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:31.823654 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:32.123671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:32.123640 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk"] Oct 30 14:32:32.127464 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:32:32.127434 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed7bb55_9c3a_4b64_b2e1_d923d8630240.slice/crio-943a122fd280d4547cc4c35a32f9fb3e90c475e48d9db7904f6cd6bc0164dc12 WatchSource:0}: Error finding container 943a122fd280d4547cc4c35a32f9fb3e90c475e48d9db7904f6cd6bc0164dc12: Status 404 returned error can't find the container with id 943a122fd280d4547cc4c35a32f9fb3e90c475e48d9db7904f6cd6bc0164dc12 Oct 30 14:32:32.149791 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:32.149758 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" event={"ID":"eed7bb55-9c3a-4b64-b2e1-d923d8630240","Type":"ContainerStarted","Data":"943a122fd280d4547cc4c35a32f9fb3e90c475e48d9db7904f6cd6bc0164dc12"} Oct 30 14:32:32.517419 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:32.517386 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f2aa45-0a1b-4ee8-9dc6-119fd1638820" path="/var/lib/kubelet/pods/76f2aa45-0a1b-4ee8-9dc6-119fd1638820/volumes" Oct 30 14:32:33.158015 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:33.157931 4968 generic.go:356] "Generic (PLEG): container finished" podID="eed7bb55-9c3a-4b64-b2e1-d923d8630240" containerID="614aa72b1a816c4f2583fda1567086c819912651444c46f0894cbafdcfa73228" exitCode=0 Oct 30 14:32:33.158165 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:33.158015 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" event={"ID":"eed7bb55-9c3a-4b64-b2e1-d923d8630240","Type":"ContainerDied","Data":"614aa72b1a816c4f2583fda1567086c819912651444c46f0894cbafdcfa73228"} Oct 30 14:32:34.303037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:34.303015 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:34.363634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:34.363605 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed7bb55-9c3a-4b64-b2e1-d923d8630240-kube-api-access-4fn7q\" (UniqueName: \"kubernetes.io/projected/eed7bb55-9c3a-4b64-b2e1-d923d8630240-kube-api-access-4fn7q\") pod \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " Oct 30 14:32:34.364008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:34.363644 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-openshift-ca\") pod \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " Oct 30 14:32:34.364008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:34.363728 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-xks-ca\") pod \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\" (UID: \"eed7bb55-9c3a-4b64-b2e1-d923d8630240\") " Oct 30 14:32:34.364147 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:34.364111 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-openshift-ca" pod "eed7bb55-9c3a-4b64-b2e1-d923d8630240" (UID: "eed7bb55-9c3a-4b64-b2e1-d923d8630240"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:34.365839 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:34.365809 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed7bb55-9c3a-4b64-b2e1-d923d8630240-kube-api-access-4fn7q" pod "eed7bb55-9c3a-4b64-b2e1-d923d8630240" (UID: "eed7bb55-9c3a-4b64-b2e1-d923d8630240"). InnerVolumeSpecName "kube-api-access-4fn7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:34.365947 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:34.365841 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-xks-ca" pod "eed7bb55-9c3a-4b64-b2e1-d923d8630240" (UID: "eed7bb55-9c3a-4b64-b2e1-d923d8630240"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:32:34.465329 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:34.465254 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:34.465329 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:34.465292 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fn7q\" (UniqueName: \"kubernetes.io/projected/eed7bb55-9c3a-4b64-b2e1-d923d8630240-kube-api-access-4fn7q\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:34.465329 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:34.465303 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed7bb55-9c3a-4b64-b2e1-d923d8630240-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:35.170863 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:35.170834 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" Oct 30 14:32:35.170863 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:35.170854 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk" event={"ID":"eed7bb55-9c3a-4b64-b2e1-d923d8630240","Type":"ContainerDied","Data":"943a122fd280d4547cc4c35a32f9fb3e90c475e48d9db7904f6cd6bc0164dc12"} Oct 30 14:32:35.171130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:35.170910 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="943a122fd280d4547cc4c35a32f9fb3e90c475e48d9db7904f6cd6bc0164dc12" Oct 30 14:32:41.030705 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:41.030669 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs"] Oct 30 14:32:41.040646 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:41.040618 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9f7c83-9xdqs"] Oct 30 14:32:42.032739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:42.032686 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs"] Oct 30 14:32:42.042394 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:42.042362 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7075a8bf-q4fxs"] Oct 30 14:32:42.518214 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:42.518179 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e15b9e-ff32-4a61-a41b-d02514acd47c" path="/var/lib/kubelet/pods/84e15b9e-ff32-4a61-a41b-d02514acd47c/volumes" Oct 30 14:32:42.518601 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:42.518588 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab82e4e2-5ac1-4904-a0ee-c9b724547a0f" path="/var/lib/kubelet/pods/ab82e4e2-5ac1-4904-a0ee-c9b724547a0f/volumes" Oct 30 14:32:43.547113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:43.547077 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:32:45.640612 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:45.640580 4968 scope.go:122] "RemoveContainer" containerID="e630481f115e90f816dce401ac5105217adc9c82ddb6215d5f1738a7cac4e4b2" Oct 30 14:32:45.651283 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:45.651260 4968 scope.go:122] "RemoveContainer" containerID="62247eb77e8560d6b8f4c4f66c0959e10d8751839f20d53e959da52b75b3707f" Oct 30 14:32:45.660828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:45.660811 4968 scope.go:122] "RemoveContainer" containerID="a40b2b020512e57f6ebe4cd0de71c7edc2195e7e25690fb59bb963807416d0cd" Oct 30 14:32:45.669059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:45.669034 4968 scope.go:122] "RemoveContainer" containerID="2bc1c0509a6bcef16c7291765403ad1fc3eb9bcb6dcd3a6bbae367e4176b6d66" Oct 30 14:32:45.677550 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:45.677533 4968 scope.go:122] "RemoveContainer" containerID="e15c2476ff44eb4123f11b592acb84d32c4cd70dc139f79ee9aaef8d360fcd6e" Oct 30 14:32:45.685479 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:45.685460 4968 scope.go:122] "RemoveContainer" containerID="5988ce60f65a43ca53cfd7f5322d8fedbb96403a1cf056334e92cce120b3c1de" Oct 30 14:32:45.693595 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:45.693572 4968 scope.go:122] "RemoveContainer" containerID="37d00805a50e7fa7a6f1fddf7621dfc0123671ef817feeee64b96b0c73aed900" Oct 30 14:32:45.701685 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:45.701671 4968 scope.go:122] "RemoveContainer" containerID="fb08b4ba2795730cbca0fb7dae668902d25e3c5da3de0075e690f753437d2d95" Oct 30 14:32:45.710968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:45.710950 4968 scope.go:122] "RemoveContainer" containerID="2846bac9b5b79b457c8901eac81f3741c455462246c6f5668f317259f9cd9793" Oct 30 14:32:47.033666 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:47.033635 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx"] Oct 30 14:32:47.044796 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:47.044768 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9de6fc-qjbsx"] Oct 30 14:32:48.033078 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:48.033045 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx"] Oct 30 14:32:48.042452 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:48.042426 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a41b3f8-lwsjx"] Oct 30 14:32:48.487471 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:48.487437 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:32:48.517406 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:48.517370 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fa2371-a482-470e-b019-baa5cd14b25c" path="/var/lib/kubelet/pods/35fa2371-a482-470e-b019-baa5cd14b25c/volumes" Oct 30 14:32:48.517798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:48.517784 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3240528-e402-43de-9702-45c2c308588f" path="/var/lib/kubelet/pods/d3240528-e402-43de-9702-45c2c308588f/volumes" Oct 30 14:32:50.444015 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:32:50.443974 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:32:55.062133 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.062097 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk"] Oct 30 14:32:55.071678 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.071649 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mtjkk"] Oct 30 14:32:55.110896 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.110865 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc"] Oct 30 14:32:55.111312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.111298 4968 cpu_manager.go:414] "containerMap: removing container" podUID="eed7bb55-9c3a-4b64-b2e1-d923d8630240" containerName="revoke-keys" Oct 30 14:32:55.111355 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.111314 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed7bb55-9c3a-4b64-b2e1-d923d8630240" containerName="revoke-keys" Oct 30 14:32:55.111409 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.111399 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed7bb55-9c3a-4b64-b2e1-d923d8630240" containerName="revoke-keys" Oct 30 14:32:55.116580 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.116564 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:55.131903 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.131883 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:32:55.132016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.131954 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:32:55.132016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.131970 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:32:55.132016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.131954 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:32:55.134624 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.134601 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc"] Oct 30 14:32:55.254924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.254890 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-589jc\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:55.255103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.254935 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-256ft\" (UniqueName: \"kubernetes.io/projected/2274d488-c756-4e6a-9822-040cac4cb7d3-kube-api-access-256ft\") pod \"maas-api-revoke-sub-3d4156f1-589jc\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:55.255103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.254955 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-589jc\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:55.356249 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.356215 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-589jc\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:55.356410 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.356260 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256ft\" (UniqueName: \"kubernetes.io/projected/2274d488-c756-4e6a-9822-040cac4cb7d3-kube-api-access-256ft\") pod \"maas-api-revoke-sub-3d4156f1-589jc\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:55.356410 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.356282 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-589jc\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:55.357022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.356998 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-589jc\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:55.358607 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.358588 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-589jc\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:55.382673 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.382646 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-256ft\" (UniqueName: \"kubernetes.io/projected/2274d488-c756-4e6a-9822-040cac4cb7d3-kube-api-access-256ft\") pod \"maas-api-revoke-sub-3d4156f1-589jc\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:55.429570 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.429546 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:55.731571 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:55.731391 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc"] Oct 30 14:32:55.734671 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:32:55.734642 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2274d488_c756_4e6a_9822_040cac4cb7d3.slice/crio-01430404be206643a4072fc6ffeb8d10aaee0d2feca4f0fbb7e46b85ebc16724 WatchSource:0}: Error finding container 01430404be206643a4072fc6ffeb8d10aaee0d2feca4f0fbb7e46b85ebc16724: Status 404 returned error can't find the container with id 01430404be206643a4072fc6ffeb8d10aaee0d2feca4f0fbb7e46b85ebc16724 Oct 30 14:32:56.297101 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:56.297013 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" event={"ID":"2274d488-c756-4e6a-9822-040cac4cb7d3","Type":"ContainerStarted","Data":"be1162a4eb5a3e0b3b24f9b4cd4003063ec56c93fa35b9edb4f7dd985c377c2b"} Oct 30 14:32:56.297101 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:56.297066 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" event={"ID":"2274d488-c756-4e6a-9822-040cac4cb7d3","Type":"ContainerStarted","Data":"01430404be206643a4072fc6ffeb8d10aaee0d2feca4f0fbb7e46b85ebc16724"} Oct 30 14:32:56.335553 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:56.335507 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" podStartSLOduration=1.335493592 podStartE2EDuration="1.335493592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:32:56.334455877 +0000 UTC m=+1571.959899261" watchObservedRunningTime="2026-10-30 14:32:56.335493592 +0000 UTC m=+1571.960936964" Oct 30 14:32:56.518101 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:56.518068 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed7bb55-9c3a-4b64-b2e1-d923d8630240" path="/var/lib/kubelet/pods/eed7bb55-9c3a-4b64-b2e1-d923d8630240/volumes" Oct 30 14:32:57.304369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:57.304333 4968 generic.go:356] "Generic (PLEG): container finished" podID="2274d488-c756-4e6a-9822-040cac4cb7d3" containerID="be1162a4eb5a3e0b3b24f9b4cd4003063ec56c93fa35b9edb4f7dd985c377c2b" exitCode=0 Oct 30 14:32:57.304845 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:57.304410 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" event={"ID":"2274d488-c756-4e6a-9822-040cac4cb7d3","Type":"ContainerDied","Data":"be1162a4eb5a3e0b3b24f9b4cd4003063ec56c93fa35b9edb4f7dd985c377c2b"} Oct 30 14:32:58.446084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:58.446060 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:58.585249 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:58.585227 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-openshift-ca\") pod \"2274d488-c756-4e6a-9822-040cac4cb7d3\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " Oct 30 14:32:58.585416 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:58.585299 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2274d488-c756-4e6a-9822-040cac4cb7d3-kube-api-access-256ft\" (UniqueName: \"kubernetes.io/projected/2274d488-c756-4e6a-9822-040cac4cb7d3-kube-api-access-256ft\") pod \"2274d488-c756-4e6a-9822-040cac4cb7d3\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " Oct 30 14:32:58.585416 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:58.585343 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-xks-ca\") pod \"2274d488-c756-4e6a-9822-040cac4cb7d3\" (UID: \"2274d488-c756-4e6a-9822-040cac4cb7d3\") " Oct 30 14:32:58.585567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:58.585545 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-openshift-ca" pod "2274d488-c756-4e6a-9822-040cac4cb7d3" (UID: "2274d488-c756-4e6a-9822-040cac4cb7d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:58.587375 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:58.587343 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-xks-ca" pod "2274d488-c756-4e6a-9822-040cac4cb7d3" (UID: "2274d488-c756-4e6a-9822-040cac4cb7d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:32:58.587494 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:58.587430 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2274d488-c756-4e6a-9822-040cac4cb7d3-kube-api-access-256ft" pod "2274d488-c756-4e6a-9822-040cac4cb7d3" (UID: "2274d488-c756-4e6a-9822-040cac4cb7d3"). InnerVolumeSpecName "kube-api-access-256ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:58.686910 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:58.686880 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.686910 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:58.686906 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-256ft\" (UniqueName: \"kubernetes.io/projected/2274d488-c756-4e6a-9822-040cac4cb7d3-kube-api-access-256ft\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.687069 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:58.686921 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2274d488-c756-4e6a-9822-040cac4cb7d3-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.316582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:59.316554 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" Oct 30 14:32:59.316773 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:59.316592 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc" event={"ID":"2274d488-c756-4e6a-9822-040cac4cb7d3","Type":"ContainerDied","Data":"01430404be206643a4072fc6ffeb8d10aaee0d2feca4f0fbb7e46b85ebc16724"} Oct 30 14:32:59.316773 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:32:59.316635 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01430404be206643a4072fc6ffeb8d10aaee0d2feca4f0fbb7e46b85ebc16724" Oct 30 14:33:07.135866 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:07.135819 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:33:12.137942 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:12.137901 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:33:17.083728 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.083676 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9"] Oct 30 14:33:17.084442 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.084419 4968 cpu_manager.go:414] "containerMap: removing container" podUID="2274d488-c756-4e6a-9822-040cac4cb7d3" containerName="revoke-keys" Oct 30 14:33:17.084491 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.084447 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2274d488-c756-4e6a-9822-040cac4cb7d3" containerName="revoke-keys" Oct 30 14:33:17.084619 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.084598 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="2274d488-c756-4e6a-9822-040cac4cb7d3" containerName="revoke-keys" Oct 30 14:33:17.088754 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.088733 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.105532 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.105507 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2c6211-openshift-default-dockercfg-4dxmw\"" Oct 30 14:33:17.115633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.115598 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9"] Oct 30 14:33:17.163648 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.163614 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-token\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.163829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.163674 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-certs\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.163829 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.163800 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-socket\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.163924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.163856 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xhx2\" (UniqueName: \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-kube-api-access-8xhx2\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.163924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.163888 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-data\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.163924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.163905 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istiod-ca-cert\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.164036 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.164007 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-envoy\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.164081 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.164058 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-podinfo\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.164118 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.164099 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-credential-socket\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.264803 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.264766 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-data\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.264803 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.264805 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istiod-ca-cert\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.265072 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.264828 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-envoy\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.265072 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.264871 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-podinfo\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.265072 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.264924 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-credential-socket\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.265072 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.264959 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-token\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.265072 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.265026 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-certs\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.265347 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.265076 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-socket\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.265347 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.265100 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xhx2\" (UniqueName: \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-kube-api-access-8xhx2\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.265347 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.265307 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-credential-socket\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.265610 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.265582 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-certs\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.265772 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.265621 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-socket\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.265958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.265933 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istiod-ca-cert\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.266078 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.266055 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-data\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.268389 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.268364 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-envoy\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.269348 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.269301 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-podinfo\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.301582 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.301539 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xhx2\" (UniqueName: \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-kube-api-access-8xhx2\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.302031 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.302002 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-token\") pod \"e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.402016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.401929 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:17.732251 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.732164 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9"] Oct 30 14:33:17.742116 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.742073 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:33:17.742209 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.742161 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:33:17.742209 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:17.742190 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:33:18.449666 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:18.449625 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" event={"ID":"a3a4b3ae-235b-4e8c-97ba-a26813153e7f","Type":"ContainerStarted","Data":"7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425"} Oct 30 14:33:18.450097 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:18.449684 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" event={"ID":"a3a4b3ae-235b-4e8c-97ba-a26813153e7f","Type":"ContainerStarted","Data":"f336787ffcee121afa5b858e1e3fe74ddcc401ac21ad7ac93cf025d9a7690860"} Oct 30 14:33:18.503290 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:18.503239 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" podStartSLOduration=1.503224503 podStartE2EDuration="1.503224503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:18.502521893 +0000 UTC m=+1594.127965267" watchObservedRunningTime="2026-10-30 14:33:18.503224503 +0000 UTC m=+1594.128667875" Oct 30 14:33:19.402255 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:19.402224 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:19.407519 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:19.407495 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:19.456157 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:19.456127 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:19.456883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:19.456865 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:20.444142 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:33:20.444109 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:33:22.782193 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:22.782162 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4"] Oct 30 14:33:22.786182 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:22.786163 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:33:22.799267 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:22.799244 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2c6211\"" Oct 30 14:33:22.808924 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:22.808898 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a395d310-dockercfg-jbh5f\"" Oct 30 14:33:22.811540 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:22.811518 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4"] Oct 30 14:33:22.882383 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:22.882349 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v"] Oct 30 14:33:22.887600 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:22.887569 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:33:22.896490 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:22.896460 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v"] Oct 30 14:33:22.932020 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:22.931990 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxvz\" (UniqueName: \"kubernetes.io/projected/bb0117f0-b7d7-43b1-aa23-b435f22d8465-kube-api-access-vrxvz\") pod \"payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4\" (UID: \"bb0117f0-b7d7-43b1-aa23-b435f22d8465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:33:22.932183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:22.932055 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0117f0-b7d7-43b1-aa23-b435f22d8465-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4\" (UID: \"bb0117f0-b7d7-43b1-aa23-b435f22d8465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:33:23.033372 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.033297 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0117f0-b7d7-43b1-aa23-b435f22d8465-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4\" (UID: \"bb0117f0-b7d7-43b1-aa23-b435f22d8465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:33:23.033372 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.033334 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v\" (UID: \"5ef5f1d6-ef08-4831-8069-027d0c1f46f8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:33:23.033566 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.033415 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjrhl\" (UniqueName: \"kubernetes.io/projected/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-kube-api-access-qjrhl\") pod \"payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v\" (UID: \"5ef5f1d6-ef08-4831-8069-027d0c1f46f8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:33:23.033566 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.033432 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxvz\" (UniqueName: \"kubernetes.io/projected/bb0117f0-b7d7-43b1-aa23-b435f22d8465-kube-api-access-vrxvz\") pod \"payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4\" (UID: \"bb0117f0-b7d7-43b1-aa23-b435f22d8465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:33:23.034008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.033989 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0117f0-b7d7-43b1-aa23-b435f22d8465-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4\" (UID: \"bb0117f0-b7d7-43b1-aa23-b435f22d8465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:33:23.054907 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.054870 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxvz\" (UniqueName: \"kubernetes.io/projected/bb0117f0-b7d7-43b1-aa23-b435f22d8465-kube-api-access-vrxvz\") pod \"payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4\" (UID: \"bb0117f0-b7d7-43b1-aa23-b435f22d8465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:33:23.100404 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.100377 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:33:23.134180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.134143 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjrhl\" (UniqueName: \"kubernetes.io/projected/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-kube-api-access-qjrhl\") pod \"payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v\" (UID: \"5ef5f1d6-ef08-4831-8069-027d0c1f46f8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:33:23.134336 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.134194 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v\" (UID: \"5ef5f1d6-ef08-4831-8069-027d0c1f46f8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:33:23.134885 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.134859 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v\" (UID: \"5ef5f1d6-ef08-4831-8069-027d0c1f46f8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:33:23.158954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.157607 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjrhl\" (UniqueName: \"kubernetes.io/projected/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-kube-api-access-qjrhl\") pod \"payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v\" (UID: \"5ef5f1d6-ef08-4831-8069-027d0c1f46f8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:33:23.210037 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.210002 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:33:23.402450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.402375 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4"] Oct 30 14:33:23.405135 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:33:23.405104 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0117f0_b7d7_43b1_aa23_b435f22d8465.slice/crio-51014ef6b992875605fe2c8bd2d6f9abed6c0b70ebc92ea70b038a66545111ac WatchSource:0}: Error finding container 51014ef6b992875605fe2c8bd2d6f9abed6c0b70ebc92ea70b038a66545111ac: Status 404 returned error can't find the container with id 51014ef6b992875605fe2c8bd2d6f9abed6c0b70ebc92ea70b038a66545111ac Oct 30 14:33:23.492263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.492228 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" event={"ID":"bb0117f0-b7d7-43b1-aa23-b435f22d8465","Type":"ContainerStarted","Data":"6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634"} Oct 30 14:33:23.492421 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.492295 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" event={"ID":"bb0117f0-b7d7-43b1-aa23-b435f22d8465","Type":"ContainerStarted","Data":"51014ef6b992875605fe2c8bd2d6f9abed6c0b70ebc92ea70b038a66545111ac"} Oct 30 14:33:23.492421 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.492330 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:33:23.519152 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.519059 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v"] Oct 30 14:33:23.521835 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:33:23.521812 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef5f1d6_ef08_4831_8069_027d0c1f46f8.slice/crio-773c98ee18382ca1be256c1f17b52d5b114dbde16a6ddc9c09fc8809ce04bc7a WatchSource:0}: Error finding container 773c98ee18382ca1be256c1f17b52d5b114dbde16a6ddc9c09fc8809ce04bc7a: Status 404 returned error can't find the container with id 773c98ee18382ca1be256c1f17b52d5b114dbde16a6ddc9c09fc8809ce04bc7a Oct 30 14:33:23.534368 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:23.534326 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" podStartSLOduration=1.534311301 podStartE2EDuration="1.534311301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:23.532302337 +0000 UTC m=+1599.157745710" watchObservedRunningTime="2026-10-30 14:33:23.534311301 +0000 UTC m=+1599.159754673" Oct 30 14:33:24.500023 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:24.499981 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" event={"ID":"5ef5f1d6-ef08-4831-8069-027d0c1f46f8","Type":"ContainerStarted","Data":"f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d"} Oct 30 14:33:24.500469 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:24.500048 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" event={"ID":"5ef5f1d6-ef08-4831-8069-027d0c1f46f8","Type":"ContainerStarted","Data":"773c98ee18382ca1be256c1f17b52d5b114dbde16a6ddc9c09fc8809ce04bc7a"} Oct 30 14:33:24.500469 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:24.500236 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:33:24.547194 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:24.547144 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" podStartSLOduration=2.547130389 podStartE2EDuration="2.547130389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:24.536551633 +0000 UTC m=+1600.161995008" watchObservedRunningTime="2026-10-30 14:33:24.547130389 +0000 UTC m=+1600.172573778" Oct 30 14:33:26.203978 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:26.203942 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:33:34.503450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:34.503419 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:33:35.497851 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:35.497819 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:33:35.498021 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:35.497883 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:33:35.509694 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:35.509668 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:33:36.603076 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:36.583940 4968 generic.go:356] "Generic (PLEG): container finished" podID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerID="7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425" exitCode=0 Oct 30 14:33:36.603076 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:36.584181 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" event={"ID":"a3a4b3ae-235b-4e8c-97ba-a26813153e7f","Type":"ContainerDied","Data":"7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425"} Oct 30 14:33:36.603076 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:36.585056 4968 scope.go:122] "RemoveContainer" containerID="7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425" Oct 30 14:33:36.603076 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:36.600912 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:33:36.603076 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:36.601009 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:33:36.603076 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:36.601055 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:33:37.610089 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:37.610051 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" event={"ID":"a3a4b3ae-235b-4e8c-97ba-a26813153e7f","Type":"ContainerStarted","Data":"280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6"} Oct 30 14:33:38.404158 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:38.404124 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:38.404502 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:38.404482 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 14:33:38.404639 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:38.404618 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 14:33:39.031753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:39.031700 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft"] Oct 30 14:33:39.043196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:39.043159 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6n9ft"] Oct 30 14:33:39.404125 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:39.404087 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 14:33:39.404309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:39.404157 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 14:33:40.403139 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:40.403098 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 14:33:40.403834 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:40.403793 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 14:33:40.410388 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:40.410335 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:33:40.517743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:40.517685 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5995f7f-0331-4a1f-aefd-6c7173c05506" path="/var/lib/kubelet/pods/e5995f7f-0331-4a1f-aefd-6c7173c05506/volumes" Oct 30 14:33:41.404087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:41.404044 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 14:33:41.404735 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:41.404681 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 14:33:42.403739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:42.403091 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 14:33:42.403739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:42.403158 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 14:33:43.406977 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:43.406947 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:43.407449 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:43.407363 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:43.408186 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:43.408160 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:33:45.768233 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:45.768199 4968 scope.go:122] "RemoveContainer" containerID="a5bdf7b7c19383bf9ff3566620a32ce50b3ab18aa1096c9fd4f4701d6ed4f96e" Oct 30 14:33:45.777312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:45.777295 4968 scope.go:122] "RemoveContainer" containerID="3d0475ac6ef5e260d63b1357312f0fd9830cc368fec643d48cd8fcfb74fd52c9" Oct 30 14:33:45.785596 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:45.785574 4968 scope.go:122] "RemoveContainer" containerID="ef6a6d0d3bbb573bdcb9a43a4e7173021c921f68dff39506c92cef3719904df7" Oct 30 14:33:45.793607 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:45.793582 4968 scope.go:122] "RemoveContainer" containerID="b74ea38d4e69e9fe820cb07a3a6eb20b1fd7660d97c76b97eb58289dadb558a0" Oct 30 14:33:45.801623 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:45.801601 4968 scope.go:122] "RemoveContainer" containerID="6470ded6b65adfaf701312774d7ea580db10c9a842ef889a1a95834f6d98fce8" Oct 30 14:33:45.809709 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:45.809691 4968 scope.go:122] "RemoveContainer" containerID="bbc7e29cfefd573c6e7347c80d1823f21327e6047a4120d505977ac64e319b90" Oct 30 14:33:45.818683 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:45.818651 4968 scope.go:122] "RemoveContainer" containerID="4d52c81e9329fb7a93a978feef1c53a045478a92c0ac3a8f2b1dafe871a134eb" Oct 30 14:33:45.827072 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:45.827046 4968 scope.go:122] "RemoveContainer" containerID="7bed2748b202e4f5922ba68807a743457741e26643f45c2186af7982e3fb5221" Oct 30 14:33:45.835263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:45.835247 4968 scope.go:122] "RemoveContainer" containerID="81edb5d3490f01cbc97d8f3646bee263c0371c9c08c78a07929c88e0a9a438d4" Oct 30 14:33:45.842982 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:45.842953 4968 scope.go:122] "RemoveContainer" containerID="6afd9fe95f41fad6d7334fda1d27f1573162e5c4462473f91db800d1688a0a29" Oct 30 14:33:45.850241 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:45.850222 4968 scope.go:122] "RemoveContainer" containerID="ea9a5c6e44253daae09b4d17e9fe7c7c3424eb4145bc73d561d011a27df0ffea" Oct 30 14:33:50.444011 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:33:50.443969 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:33:55.561600 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:33:55.561561 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:34:01.618572 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.618537 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf"] Oct 30 14:34:01.624297 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.624275 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:01.646351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.646309 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf"] Oct 30 14:34:01.650970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.650570 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:34:01.650970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.650604 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:34:01.650970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.650693 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:34:01.650970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.650740 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:34:01.761605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.761576 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn5rb\" (UniqueName: \"kubernetes.io/projected/903385c8-a558-448f-aac3-5e7f57d8d05b-kube-api-access-bn5rb\") pod \"maas-api-revoke-sub-5b0fe258-s6cbf\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:01.761793 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.761612 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b0fe258-s6cbf\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:01.761793 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.761703 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b0fe258-s6cbf\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:01.862767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.862730 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5rb\" (UniqueName: \"kubernetes.io/projected/903385c8-a558-448f-aac3-5e7f57d8d05b-kube-api-access-bn5rb\") pod \"maas-api-revoke-sub-5b0fe258-s6cbf\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:01.862767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.862772 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b0fe258-s6cbf\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:01.862987 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.862819 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b0fe258-s6cbf\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:01.863470 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.863442 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b0fe258-s6cbf\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:01.865331 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.865307 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b0fe258-s6cbf\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:01.885071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.885011 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5rb\" (UniqueName: \"kubernetes.io/projected/903385c8-a558-448f-aac3-5e7f57d8d05b-kube-api-access-bn5rb\") pod \"maas-api-revoke-sub-5b0fe258-s6cbf\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:01.937559 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:01.937537 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:02.237944 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:02.237814 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf"] Oct 30 14:34:02.240582 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:34:02.240553 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903385c8_a558_448f_aac3_5e7f57d8d05b.slice/crio-60d56e35d11b860ac1b5551c2488b3306438f3b65fd67010ea04280764fa9b57 WatchSource:0}: Error finding container 60d56e35d11b860ac1b5551c2488b3306438f3b65fd67010ea04280764fa9b57: Status 404 returned error can't find the container with id 60d56e35d11b860ac1b5551c2488b3306438f3b65fd67010ea04280764fa9b57 Oct 30 14:34:02.775071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:02.774978 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" event={"ID":"903385c8-a558-448f-aac3-5e7f57d8d05b","Type":"ContainerStarted","Data":"cefe0d43f73523cc8f4d09223a045cd66e112fd04cb079fc0b6043a4fa1c90ed"} Oct 30 14:34:02.775071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:02.775037 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" event={"ID":"903385c8-a558-448f-aac3-5e7f57d8d05b","Type":"ContainerStarted","Data":"60d56e35d11b860ac1b5551c2488b3306438f3b65fd67010ea04280764fa9b57"} Oct 30 14:34:02.813869 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:02.813820 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" podStartSLOduration=1.8138059549999999 podStartE2EDuration="1.813805955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:02.812843004 +0000 UTC m=+1638.438286379" watchObservedRunningTime="2026-10-30 14:34:02.813805955 +0000 UTC m=+1638.439249327" Oct 30 14:34:03.783908 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:03.783870 4968 generic.go:356] "Generic (PLEG): container finished" podID="903385c8-a558-448f-aac3-5e7f57d8d05b" containerID="cefe0d43f73523cc8f4d09223a045cd66e112fd04cb079fc0b6043a4fa1c90ed" exitCode=0 Oct 30 14:34:03.784381 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:03.783945 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" event={"ID":"903385c8-a558-448f-aac3-5e7f57d8d05b","Type":"ContainerDied","Data":"cefe0d43f73523cc8f4d09223a045cd66e112fd04cb079fc0b6043a4fa1c90ed"} Oct 30 14:34:04.927693 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:04.927670 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:05.096339 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.096309 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903385c8-a558-448f-aac3-5e7f57d8d05b-kube-api-access-bn5rb\" (UniqueName: \"kubernetes.io/projected/903385c8-a558-448f-aac3-5e7f57d8d05b-kube-api-access-bn5rb\") pod \"903385c8-a558-448f-aac3-5e7f57d8d05b\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " Oct 30 14:34:05.096506 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.096357 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-openshift-ca\") pod \"903385c8-a558-448f-aac3-5e7f57d8d05b\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " Oct 30 14:34:05.096506 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.096458 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-xks-ca\") pod \"903385c8-a558-448f-aac3-5e7f57d8d05b\" (UID: \"903385c8-a558-448f-aac3-5e7f57d8d05b\") " Oct 30 14:34:05.096800 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.096775 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-openshift-ca" pod "903385c8-a558-448f-aac3-5e7f57d8d05b" (UID: "903385c8-a558-448f-aac3-5e7f57d8d05b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:05.098549 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.098518 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-xks-ca" pod "903385c8-a558-448f-aac3-5e7f57d8d05b" (UID: "903385c8-a558-448f-aac3-5e7f57d8d05b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:34:05.098549 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.098536 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903385c8-a558-448f-aac3-5e7f57d8d05b-kube-api-access-bn5rb" pod "903385c8-a558-448f-aac3-5e7f57d8d05b" (UID: "903385c8-a558-448f-aac3-5e7f57d8d05b"). InnerVolumeSpecName "kube-api-access-bn5rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:05.197535 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.197498 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn5rb\" (UniqueName: \"kubernetes.io/projected/903385c8-a558-448f-aac3-5e7f57d8d05b-kube-api-access-bn5rb\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:05.197535 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.197530 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:05.197535 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.197540 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903385c8-a558-448f-aac3-5e7f57d8d05b-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:05.796085 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.796057 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" Oct 30 14:34:05.796258 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.796052 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf" event={"ID":"903385c8-a558-448f-aac3-5e7f57d8d05b","Type":"ContainerDied","Data":"60d56e35d11b860ac1b5551c2488b3306438f3b65fd67010ea04280764fa9b57"} Oct 30 14:34:05.796258 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:05.796226 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60d56e35d11b860ac1b5551c2488b3306438f3b65fd67010ea04280764fa9b57" Oct 30 14:34:12.784667 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.784627 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr"] Oct 30 14:34:12.785209 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.785140 4968 cpu_manager.go:414] "containerMap: removing container" podUID="903385c8-a558-448f-aac3-5e7f57d8d05b" containerName="revoke-keys" Oct 30 14:34:12.785209 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.785153 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903385c8-a558-448f-aac3-5e7f57d8d05b" containerName="revoke-keys" Oct 30 14:34:12.785336 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.785261 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="903385c8-a558-448f-aac3-5e7f57d8d05b" containerName="revoke-keys" Oct 30 14:34:12.788641 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.788618 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:12.809525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.809495 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:34:12.809690 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.809557 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:34:12.823281 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.823252 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:34:12.823457 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.823391 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:34:12.825573 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.825544 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr"] Oct 30 14:34:12.869318 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.869283 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:12.869480 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.869332 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:12.869480 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.869449 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rn6k\" (UniqueName: \"kubernetes.io/projected/aa0e889a-09bf-415c-8021-004411c91cf2-kube-api-access-9rn6k\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:12.970048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.970010 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:12.970253 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.970074 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:12.970253 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.970141 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rn6k\" (UniqueName: \"kubernetes.io/projected/aa0e889a-09bf-415c-8021-004411c91cf2-kube-api-access-9rn6k\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:12.970677 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.970651 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:12.972413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.972389 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:12.998079 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:12.998051 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rn6k\" (UniqueName: \"kubernetes.io/projected/aa0e889a-09bf-415c-8021-004411c91cf2-kube-api-access-9rn6k\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:13.082916 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:13.082882 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:34:13.102514 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:13.102482 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:13.407110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:13.407074 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr"] Oct 30 14:34:13.407110 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:34:13.407097 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0e889a_09bf_415c_8021_004411c91cf2.slice/crio-1a5ade15e138fc044eda98c3594463123c62ca95915f3d11f3258f4cc617fb7f WatchSource:0}: Error finding container 1a5ade15e138fc044eda98c3594463123c62ca95915f3d11f3258f4cc617fb7f: Status 404 returned error can't find the container with id 1a5ade15e138fc044eda98c3594463123c62ca95915f3d11f3258f4cc617fb7f Oct 30 14:34:13.845036 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:13.845001 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" event={"ID":"aa0e889a-09bf-415c-8021-004411c91cf2","Type":"ContainerStarted","Data":"472aae7f9dcd4bbca37cbae666cb75db3415b74fb3df2b972232ba334e1b8c48"} Oct 30 14:34:13.845433 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:13.845056 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" event={"ID":"aa0e889a-09bf-415c-8021-004411c91cf2","Type":"ContainerStarted","Data":"1a5ade15e138fc044eda98c3594463123c62ca95915f3d11f3258f4cc617fb7f"} Oct 30 14:34:13.934262 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:13.934209 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" podStartSLOduration=1.934195516 podStartE2EDuration="1.934195516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:13.934046862 +0000 UTC m=+1649.559490236" watchObservedRunningTime="2026-10-30 14:34:13.934195516 +0000 UTC m=+1649.559638890" Oct 30 14:34:14.852681 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:14.852646 4968 generic.go:356] "Generic (PLEG): container finished" podID="aa0e889a-09bf-415c-8021-004411c91cf2" containerID="472aae7f9dcd4bbca37cbae666cb75db3415b74fb3df2b972232ba334e1b8c48" exitCode=0 Oct 30 14:34:14.853110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:14.852755 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" event={"ID":"aa0e889a-09bf-415c-8021-004411c91cf2","Type":"ContainerDied","Data":"472aae7f9dcd4bbca37cbae666cb75db3415b74fb3df2b972232ba334e1b8c48"} Oct 30 14:34:16.012920 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.012897 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:16.098899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.098864 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0e889a-09bf-415c-8021-004411c91cf2-kube-api-access-9rn6k\" (UniqueName: \"kubernetes.io/projected/aa0e889a-09bf-415c-8021-004411c91cf2-kube-api-access-9rn6k\") pod \"aa0e889a-09bf-415c-8021-004411c91cf2\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " Oct 30 14:34:16.099053 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.098963 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-openshift-ca\") pod \"aa0e889a-09bf-415c-8021-004411c91cf2\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " Oct 30 14:34:16.099053 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.099000 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-xks-ca\") pod \"aa0e889a-09bf-415c-8021-004411c91cf2\" (UID: \"aa0e889a-09bf-415c-8021-004411c91cf2\") " Oct 30 14:34:16.099330 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.099306 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-openshift-ca" pod "aa0e889a-09bf-415c-8021-004411c91cf2" (UID: "aa0e889a-09bf-415c-8021-004411c91cf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:16.101135 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.101101 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-xks-ca" pod "aa0e889a-09bf-415c-8021-004411c91cf2" (UID: "aa0e889a-09bf-415c-8021-004411c91cf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:34:16.101135 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.101110 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0e889a-09bf-415c-8021-004411c91cf2-kube-api-access-9rn6k" pod "aa0e889a-09bf-415c-8021-004411c91cf2" (UID: "aa0e889a-09bf-415c-8021-004411c91cf2"). InnerVolumeSpecName "kube-api-access-9rn6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:16.199828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.199792 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:16.199828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.199821 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa0e889a-09bf-415c-8021-004411c91cf2-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:16.199828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.199830 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rn6k\" (UniqueName: \"kubernetes.io/projected/aa0e889a-09bf-415c-8021-004411c91cf2-kube-api-access-9rn6k\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:16.865501 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.865466 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" event={"ID":"aa0e889a-09bf-415c-8021-004411c91cf2","Type":"ContainerDied","Data":"1a5ade15e138fc044eda98c3594463123c62ca95915f3d11f3258f4cc617fb7f"} Oct 30 14:34:16.865501 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.865498 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr" Oct 30 14:34:16.865743 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:16.865518 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a5ade15e138fc044eda98c3594463123c62ca95915f3d11f3258f4cc617fb7f" Oct 30 14:34:20.444078 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:34:20.444042 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:34:23.267970 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:23.267933 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v"] Oct 30 14:34:23.285267 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:23.285237 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4"] Oct 30 14:34:24.870414 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:24.870375 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr"] Oct 30 14:34:24.885507 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:24.885474 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c6211-k29jr"] Oct 30 14:34:26.518493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:26.518412 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0e889a-09bf-415c-8021-004411c91cf2" path="/var/lib/kubelet/pods/aa0e889a-09bf-415c-8021-004411c91cf2/volumes" Oct 30 14:34:26.953084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:26.953047 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:34:28.038110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:28.038073 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw"] Oct 30 14:34:28.049848 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:28.049821 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9lbjw"] Oct 30 14:34:28.268656 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:28.268594 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" podUID="5ef5f1d6-ef08-4831-8069-027d0c1f46f8" containerName="payload-processing" containerID="cri-o://f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d" gracePeriod=25 Oct 30 14:34:28.285763 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:28.285726 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" podUID="bb0117f0-b7d7-43b1-aa23-b435f22d8465" containerName="payload-pre-processing" containerID="cri-o://6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634" gracePeriod=25 Oct 30 14:34:28.517536 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:28.517496 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cfca8db-ef2e-46c1-ba78-b787ddff44c9" path="/var/lib/kubelet/pods/7cfca8db-ef2e-46c1-ba78-b787ddff44c9/volumes" Oct 30 14:34:32.116699 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:32.116669 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9"] Oct 30 14:34:32.117266 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:32.116934 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" containerID="cri-o://280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6" gracePeriod=30 Oct 30 14:34:33.408199 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.408159 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc"] Oct 30 14:34:33.418616 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.418593 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-589jc"] Oct 30 14:34:33.444077 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.444050 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587"] Oct 30 14:34:33.444758 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.444739 4968 cpu_manager.go:414] "containerMap: removing container" podUID="aa0e889a-09bf-415c-8021-004411c91cf2" containerName="revoke-keys" Oct 30 14:34:33.444858 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.444759 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0e889a-09bf-415c-8021-004411c91cf2" containerName="revoke-keys" Oct 30 14:34:33.444937 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.444897 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0e889a-09bf-415c-8021-004411c91cf2" containerName="revoke-keys" Oct 30 14:34:33.449755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.449733 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:33.456236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.456052 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4w587\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:33.456236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.456094 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4w587\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:33.456361 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.456338 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4gxf\" (UniqueName: \"kubernetes.io/projected/202c76db-19bc-4c08-8071-8cff9dda8de6-kube-api-access-b4gxf\") pod \"maas-api-revoke-sub-3d4156f1-4w587\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:33.465121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.465098 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:34:33.465247 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.465137 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:34:33.465247 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.465182 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:34:33.465361 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.465253 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:34:33.466950 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.466928 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587"] Oct 30 14:34:33.557769 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.557736 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4w587\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:33.557955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.557781 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4w587\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:33.557955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.557866 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4gxf\" (UniqueName: \"kubernetes.io/projected/202c76db-19bc-4c08-8071-8cff9dda8de6-kube-api-access-b4gxf\") pod \"maas-api-revoke-sub-3d4156f1-4w587\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:33.558438 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.558413 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4w587\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:33.560246 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.560226 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4w587\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:33.582039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.582009 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4gxf\" (UniqueName: \"kubernetes.io/projected/202c76db-19bc-4c08-8071-8cff9dda8de6-kube-api-access-b4gxf\") pod \"maas-api-revoke-sub-3d4156f1-4w587\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:33.762606 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:33.762523 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:34.066596 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:34.066572 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587"] Oct 30 14:34:34.068234 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:34:34.068206 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202c76db_19bc_4c08_8071_8cff9dda8de6.slice/crio-7029b1fe210ab8f36f346d2e05d602e3c0ff619870b6b2ead3ba5e86ebba1a36 WatchSource:0}: Error finding container 7029b1fe210ab8f36f346d2e05d602e3c0ff619870b6b2ead3ba5e86ebba1a36: Status 404 returned error can't find the container with id 7029b1fe210ab8f36f346d2e05d602e3c0ff619870b6b2ead3ba5e86ebba1a36 Oct 30 14:34:34.501382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:34.501345 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" podUID="bb0117f0-b7d7-43b1-aa23-b435f22d8465" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:34:34.518247 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:34.518213 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2274d488-c756-4e6a-9822-040cac4cb7d3" path="/var/lib/kubelet/pods/2274d488-c756-4e6a-9822-040cac4cb7d3/volumes" Oct 30 14:34:35.000180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:35.000147 4968 generic.go:356] "Generic (PLEG): container finished" podID="202c76db-19bc-4c08-8071-8cff9dda8de6" containerID="7e77768f0671289a0f00523d473d524d741e7db903dad6816c63563659f8b238" exitCode=0 Oct 30 14:34:35.000180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:35.000182 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" event={"ID":"202c76db-19bc-4c08-8071-8cff9dda8de6","Type":"ContainerDied","Data":"7e77768f0671289a0f00523d473d524d741e7db903dad6816c63563659f8b238"} Oct 30 14:34:35.000411 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:35.000217 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" event={"ID":"202c76db-19bc-4c08-8071-8cff9dda8de6","Type":"ContainerStarted","Data":"7029b1fe210ab8f36f346d2e05d602e3c0ff619870b6b2ead3ba5e86ebba1a36"} Oct 30 14:34:35.506971 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:35.506935 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" podUID="5ef5f1d6-ef08-4831-8069-027d0c1f46f8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:34:36.140218 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:36.140191 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:36.181274 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:36.181250 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-openshift-ca\") pod \"202c76db-19bc-4c08-8071-8cff9dda8de6\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " Oct 30 14:34:36.181427 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:36.181332 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-xks-ca\") pod \"202c76db-19bc-4c08-8071-8cff9dda8de6\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " Oct 30 14:34:36.181427 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:36.181390 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202c76db-19bc-4c08-8071-8cff9dda8de6-kube-api-access-b4gxf\" (UniqueName: \"kubernetes.io/projected/202c76db-19bc-4c08-8071-8cff9dda8de6-kube-api-access-b4gxf\") pod \"202c76db-19bc-4c08-8071-8cff9dda8de6\" (UID: \"202c76db-19bc-4c08-8071-8cff9dda8de6\") " Oct 30 14:34:36.181915 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:36.181888 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-openshift-ca" pod "202c76db-19bc-4c08-8071-8cff9dda8de6" (UID: "202c76db-19bc-4c08-8071-8cff9dda8de6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:36.183907 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:36.183880 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-xks-ca" pod "202c76db-19bc-4c08-8071-8cff9dda8de6" (UID: "202c76db-19bc-4c08-8071-8cff9dda8de6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:34:36.184113 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:36.184089 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202c76db-19bc-4c08-8071-8cff9dda8de6-kube-api-access-b4gxf" pod "202c76db-19bc-4c08-8071-8cff9dda8de6" (UID: "202c76db-19bc-4c08-8071-8cff9dda8de6"). InnerVolumeSpecName "kube-api-access-b4gxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:36.282933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:36.282862 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:36.282933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:36.282887 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202c76db-19bc-4c08-8071-8cff9dda8de6-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:36.282933 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:36.282898 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4gxf\" (UniqueName: \"kubernetes.io/projected/202c76db-19bc-4c08-8071-8cff9dda8de6-kube-api-access-b4gxf\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:37.013338 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.013303 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" event={"ID":"202c76db-19bc-4c08-8071-8cff9dda8de6","Type":"ContainerDied","Data":"7029b1fe210ab8f36f346d2e05d602e3c0ff619870b6b2ead3ba5e86ebba1a36"} Oct 30 14:34:37.013338 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.013328 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587" Oct 30 14:34:37.013825 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.013353 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7029b1fe210ab8f36f346d2e05d602e3c0ff619870b6b2ead3ba5e86ebba1a36" Oct 30 14:34:37.369009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.368984 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:34:37.392819 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.392797 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-podinfo\") pod \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " Oct 30 14:34:37.392925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.392843 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-kube-api-access-8xhx2\" (UniqueName: \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-kube-api-access-8xhx2\") pod \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " Oct 30 14:34:37.392925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.392877 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-credential-socket\") pod \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " Oct 30 14:34:37.392925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.392898 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-envoy\") pod \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " Oct 30 14:34:37.393075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.392931 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-token\" (UniqueName: \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-token\") pod \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " Oct 30 14:34:37.393075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.392956 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-data\") pod \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " Oct 30 14:34:37.393075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.393005 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istiod-ca-cert\") pod \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " Oct 30 14:34:37.393236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.393074 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-socket\") pod \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " Oct 30 14:34:37.393236 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.393137 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-certs\") pod \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\" (UID: \"a3a4b3ae-235b-4e8c-97ba-a26813153e7f\") " Oct 30 14:34:37.393362 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.393217 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-credential-socket" pod "a3a4b3ae-235b-4e8c-97ba-a26813153e7f" (UID: "a3a4b3ae-235b-4e8c-97ba-a26813153e7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:34:37.393435 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.393394 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-certs" pod "a3a4b3ae-235b-4e8c-97ba-a26813153e7f" (UID: "a3a4b3ae-235b-4e8c-97ba-a26813153e7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:34:37.393565 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.393518 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:37.393565 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.393539 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:37.393741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.393580 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-data" pod "a3a4b3ae-235b-4e8c-97ba-a26813153e7f" (UID: "a3a4b3ae-235b-4e8c-97ba-a26813153e7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:34:37.393741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.393588 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-socket" pod "a3a4b3ae-235b-4e8c-97ba-a26813153e7f" (UID: "a3a4b3ae-235b-4e8c-97ba-a26813153e7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:34:37.393741 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.393634 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istiod-ca-cert" pod "a3a4b3ae-235b-4e8c-97ba-a26813153e7f" (UID: "a3a4b3ae-235b-4e8c-97ba-a26813153e7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:37.394961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.394942 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-podinfo" pod "a3a4b3ae-235b-4e8c-97ba-a26813153e7f" (UID: "a3a4b3ae-235b-4e8c-97ba-a26813153e7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:34:37.395110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.395091 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-token" pod "a3a4b3ae-235b-4e8c-97ba-a26813153e7f" (UID: "a3a4b3ae-235b-4e8c-97ba-a26813153e7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:37.395442 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.395419 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-envoy" pod "a3a4b3ae-235b-4e8c-97ba-a26813153e7f" (UID: "a3a4b3ae-235b-4e8c-97ba-a26813153e7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:34:37.395525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.395438 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-kube-api-access-8xhx2" pod "a3a4b3ae-235b-4e8c-97ba-a26813153e7f" (UID: "a3a4b3ae-235b-4e8c-97ba-a26813153e7f"). InnerVolumeSpecName "kube-api-access-8xhx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:37.495052 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.495020 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:37.495052 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.495047 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xhx2\" (UniqueName: \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-kube-api-access-8xhx2\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:37.495052 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.495057 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:37.495267 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.495065 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:37.495267 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.495073 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:37.495267 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.495080 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:37.495267 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:37.495088 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a4b3ae-235b-4e8c-97ba-a26813153e7f-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:38.021182 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.021148 4968 generic.go:356] "Generic (PLEG): container finished" podID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerID="280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6" exitCode=0 Oct 30 14:34:38.021685 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.021236 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" Oct 30 14:34:38.021685 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.021229 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" event={"ID":"a3a4b3ae-235b-4e8c-97ba-a26813153e7f","Type":"ContainerDied","Data":"280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6"} Oct 30 14:34:38.021685 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.021364 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9" event={"ID":"a3a4b3ae-235b-4e8c-97ba-a26813153e7f","Type":"ContainerDied","Data":"f336787ffcee121afa5b858e1e3fe74ddcc401ac21ad7ac93cf025d9a7690860"} Oct 30 14:34:38.021685 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.021386 4968 scope.go:122] "RemoveContainer" containerID="280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6" Oct 30 14:34:38.033413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.033392 4968 scope.go:122] "RemoveContainer" containerID="7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425" Oct 30 14:34:38.041994 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.041978 4968 scope.go:122] "RemoveContainer" containerID="280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6" Oct 30 14:34:38.042279 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:34:38.042258 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6\": container with ID starting with 280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6 not found: ID does not exist" containerID="280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6" Oct 30 14:34:38.042323 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.042288 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6"} err="failed to get container status \"280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6\": rpc error: code = NotFound desc = could not find container \"280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6\": container with ID starting with 280b051632efff75ba9c1262cab60629c7894b0e566778325a72f2f11eda06f6 not found: ID does not exist" Oct 30 14:34:38.042323 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.042306 4968 scope.go:122] "RemoveContainer" containerID="7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425" Oct 30 14:34:38.042520 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:34:38.042503 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425\": container with ID starting with 7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425 not found: ID does not exist" containerID="7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425" Oct 30 14:34:38.042562 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.042527 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425"} err="failed to get container status \"7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425\": rpc error: code = NotFound desc = could not find container \"7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425\": container with ID starting with 7985b6ebb0f5db0fb0cacf14078c6c9c22f63e1c6c07d48cf5066cb8b2220425 not found: ID does not exist" Oct 30 14:34:38.086682 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.086652 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9"] Oct 30 14:34:38.098686 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.098668 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c6211-openshift-default-5d4d54dfbfk5fn9"] Oct 30 14:34:38.518352 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:38.518310 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" path="/var/lib/kubelet/pods/a3a4b3ae-235b-4e8c-97ba-a26813153e7f/volumes" Oct 30 14:34:44.502261 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:44.502225 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" podUID="bb0117f0-b7d7-43b1-aa23-b435f22d8465" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:34:45.477176 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:45.477145 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:34:45.507346 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:45.507298 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" podUID="5ef5f1d6-ef08-4831-8069-027d0c1f46f8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:34:45.938773 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:45.932444 4968 scope.go:122] "RemoveContainer" containerID="20083988a4630024a806ec29bb9e7509e566d1f2f82fcb941b22cb3f7a6a318f" Oct 30 14:34:48.548163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.548138 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:34:48.578086 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.578064 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:34:48.648963 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.648936 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0117f0-b7d7-43b1-aa23-b435f22d8465-kube-api-access-vrxvz\" (UniqueName: \"kubernetes.io/projected/bb0117f0-b7d7-43b1-aa23-b435f22d8465-kube-api-access-vrxvz\") pod \"bb0117f0-b7d7-43b1-aa23-b435f22d8465\" (UID: \"bb0117f0-b7d7-43b1-aa23-b435f22d8465\") " Oct 30 14:34:48.649118 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.649004 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-kube-api-access-qjrhl\" (UniqueName: \"kubernetes.io/projected/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-kube-api-access-qjrhl\") pod \"5ef5f1d6-ef08-4831-8069-027d0c1f46f8\" (UID: \"5ef5f1d6-ef08-4831-8069-027d0c1f46f8\") " Oct 30 14:34:48.649178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.649116 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-plugins-config-volume\") pod \"5ef5f1d6-ef08-4831-8069-027d0c1f46f8\" (UID: \"5ef5f1d6-ef08-4831-8069-027d0c1f46f8\") " Oct 30 14:34:48.649247 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.649203 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0117f0-b7d7-43b1-aa23-b435f22d8465-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0117f0-b7d7-43b1-aa23-b435f22d8465-plugins-config-volume\") pod \"bb0117f0-b7d7-43b1-aa23-b435f22d8465\" (UID: \"bb0117f0-b7d7-43b1-aa23-b435f22d8465\") " Oct 30 14:34:48.649520 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.649481 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-plugins-config-volume" pod "5ef5f1d6-ef08-4831-8069-027d0c1f46f8" (UID: "5ef5f1d6-ef08-4831-8069-027d0c1f46f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:48.650014 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.649587 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0117f0-b7d7-43b1-aa23-b435f22d8465-plugins-config-volume" pod "bb0117f0-b7d7-43b1-aa23-b435f22d8465" (UID: "bb0117f0-b7d7-43b1-aa23-b435f22d8465"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:48.651104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.651010 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-kube-api-access-qjrhl" pod "5ef5f1d6-ef08-4831-8069-027d0c1f46f8" (UID: "5ef5f1d6-ef08-4831-8069-027d0c1f46f8"). InnerVolumeSpecName "kube-api-access-qjrhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:48.651104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.651083 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0117f0-b7d7-43b1-aa23-b435f22d8465-kube-api-access-vrxvz" pod "bb0117f0-b7d7-43b1-aa23-b435f22d8465" (UID: "bb0117f0-b7d7-43b1-aa23-b435f22d8465"). InnerVolumeSpecName "kube-api-access-vrxvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:48.750301 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.750260 4968 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0117f0-b7d7-43b1-aa23-b435f22d8465-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:48.750301 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.750301 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrxvz\" (UniqueName: \"kubernetes.io/projected/bb0117f0-b7d7-43b1-aa23-b435f22d8465-kube-api-access-vrxvz\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:48.750496 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.750316 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjrhl\" (UniqueName: \"kubernetes.io/projected/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-kube-api-access-qjrhl\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:48.750496 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:48.750329 4968 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef5f1d6-ef08-4831-8069-027d0c1f46f8-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:49.037640 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.037607 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7"] Oct 30 14:34:49.040184 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.040161 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b77p7"] Oct 30 14:34:49.116213 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.116181 4968 generic.go:356] "Generic (PLEG): container finished" podID="5ef5f1d6-ef08-4831-8069-027d0c1f46f8" containerID="f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d" exitCode=0 Oct 30 14:34:49.116404 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.116231 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" Oct 30 14:34:49.116404 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.116263 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" event={"ID":"5ef5f1d6-ef08-4831-8069-027d0c1f46f8","Type":"ContainerDied","Data":"f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d"} Oct 30 14:34:49.116404 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.116321 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v" event={"ID":"5ef5f1d6-ef08-4831-8069-027d0c1f46f8","Type":"ContainerDied","Data":"773c98ee18382ca1be256c1f17b52d5b114dbde16a6ddc9c09fc8809ce04bc7a"} Oct 30 14:34:49.116404 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.116350 4968 scope.go:122] "RemoveContainer" containerID="f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d" Oct 30 14:34:49.118213 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.118176 4968 generic.go:356] "Generic (PLEG): container finished" podID="bb0117f0-b7d7-43b1-aa23-b435f22d8465" containerID="6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634" exitCode=0 Oct 30 14:34:49.118351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.118228 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" event={"ID":"bb0117f0-b7d7-43b1-aa23-b435f22d8465","Type":"ContainerDied","Data":"6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634"} Oct 30 14:34:49.118351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.118263 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" event={"ID":"bb0117f0-b7d7-43b1-aa23-b435f22d8465","Type":"ContainerDied","Data":"51014ef6b992875605fe2c8bd2d6f9abed6c0b70ebc92ea70b038a66545111ac"} Oct 30 14:34:49.118351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.118319 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4" Oct 30 14:34:49.129132 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.128976 4968 scope.go:122] "RemoveContainer" containerID="f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d" Oct 30 14:34:49.129394 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:34:49.129370 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d\": container with ID starting with f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d not found: ID does not exist" containerID="f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d" Oct 30 14:34:49.129485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.129406 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d"} err="failed to get container status \"f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d\": rpc error: code = NotFound desc = could not find container \"f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d\": container with ID starting with f864863c1940c13c987c61a7c987c88a5d44546ef5e6aa92eb7c2cbf7b0b894d not found: ID does not exist" Oct 30 14:34:49.129485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.129430 4968 scope.go:122] "RemoveContainer" containerID="6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634" Oct 30 14:34:49.137911 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.137892 4968 scope.go:122] "RemoveContainer" containerID="6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634" Oct 30 14:34:49.138168 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:34:49.138150 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634\": container with ID starting with 6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634 not found: ID does not exist" containerID="6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634" Oct 30 14:34:49.138214 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.138178 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634"} err="failed to get container status \"6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634\": rpc error: code = NotFound desc = could not find container \"6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634\": container with ID starting with 6ff158c62c15af75ec0047498ce14a1f62abc6fcaf6570f4cd89ca56d4f46634 not found: ID does not exist" Oct 30 14:34:49.168471 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.168444 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v"] Oct 30 14:34:49.173819 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.173797 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c6211-7558c5d55vhd6v"] Oct 30 14:34:49.206971 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.206951 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4"] Oct 30 14:34:49.213682 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:49.213663 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c6211-5d6c8bqtv4"] Oct 30 14:34:50.443945 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:34:50.443906 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:34:50.520392 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:50.520357 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef5f1d6-ef08-4831-8069-027d0c1f46f8" path="/var/lib/kubelet/pods/5ef5f1d6-ef08-4831-8069-027d0c1f46f8/volumes" Oct 30 14:34:50.520785 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:50.520771 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01e9983-027b-450f-bc1c-a3764d429dd7" path="/var/lib/kubelet/pods/b01e9983-027b-450f-bc1c-a3764d429dd7/volumes" Oct 30 14:34:50.521103 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:50.521092 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0117f0-b7d7-43b1-aa23-b435f22d8465" path="/var/lib/kubelet/pods/bb0117f0-b7d7-43b1-aa23-b435f22d8465/volumes" Oct 30 14:34:51.252019 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:51.251987 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:34:57.503172 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:57.503134 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs"] Oct 30 14:34:57.503591 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:57.503371 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" podUID="be2a30d6-be3f-4247-8223-714a89309f07" containerName="manager" containerID="cri-o://74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa" gracePeriod=10 Oct 30 14:34:57.760541 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:57.760481 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:34:57.932350 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:57.932312 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2a30d6-be3f-4247-8223-714a89309f07-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be2a30d6-be3f-4247-8223-714a89309f07-extensions-socket-volume\") pod \"be2a30d6-be3f-4247-8223-714a89309f07\" (UID: \"be2a30d6-be3f-4247-8223-714a89309f07\") " Oct 30 14:34:57.932547 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:57.932375 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2a30d6-be3f-4247-8223-714a89309f07-kube-api-access-nvxk5\" (UniqueName: \"kubernetes.io/projected/be2a30d6-be3f-4247-8223-714a89309f07-kube-api-access-nvxk5\") pod \"be2a30d6-be3f-4247-8223-714a89309f07\" (UID: \"be2a30d6-be3f-4247-8223-714a89309f07\") " Oct 30 14:34:57.932759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:57.932735 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2a30d6-be3f-4247-8223-714a89309f07-extensions-socket-volume" pod "be2a30d6-be3f-4247-8223-714a89309f07" (UID: "be2a30d6-be3f-4247-8223-714a89309f07"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:34:57.934449 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:57.934424 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2a30d6-be3f-4247-8223-714a89309f07-kube-api-access-nvxk5" pod "be2a30d6-be3f-4247-8223-714a89309f07" (UID: "be2a30d6-be3f-4247-8223-714a89309f07"). InnerVolumeSpecName "kube-api-access-nvxk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:58.033401 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.033296 4968 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be2a30d6-be3f-4247-8223-714a89309f07-extensions-socket-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:58.033401 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.033324 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvxk5\" (UniqueName: \"kubernetes.io/projected/be2a30d6-be3f-4247-8223-714a89309f07-kube-api-access-nvxk5\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:34:58.202981 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.202943 4968 generic.go:356] "Generic (PLEG): container finished" podID="be2a30d6-be3f-4247-8223-714a89309f07" containerID="74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa" exitCode=0 Oct 30 14:34:58.203174 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.202993 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" Oct 30 14:34:58.203174 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.203031 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" event={"ID":"be2a30d6-be3f-4247-8223-714a89309f07","Type":"ContainerDied","Data":"74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa"} Oct 30 14:34:58.203174 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.203099 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs" event={"ID":"be2a30d6-be3f-4247-8223-714a89309f07","Type":"ContainerDied","Data":"7c3e088423f1abca82498a28f7df1f3f0df1b3e1bf2943ccaf82aef2a9a2b66b"} Oct 30 14:34:58.203174 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.203129 4968 scope.go:122] "RemoveContainer" containerID="74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa" Oct 30 14:34:58.214087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.214068 4968 scope.go:122] "RemoveContainer" containerID="74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa" Oct 30 14:34:58.214324 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:34:58.214305 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa\": container with ID starting with 74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa not found: ID does not exist" containerID="74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa" Oct 30 14:34:58.214367 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.214333 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa"} err="failed to get container status \"74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa\": rpc error: code = NotFound desc = could not find container \"74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa\": container with ID starting with 74a09a2215e1c8f6e19035779aeb086ca0403272768bbfb2f2927b3736511efa not found: ID does not exist" Oct 30 14:34:58.256554 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.256524 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs"] Oct 30 14:34:58.268542 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.268510 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-q79hs"] Oct 30 14:34:58.517190 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:34:58.517155 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2a30d6-be3f-4247-8223-714a89309f07" path="/var/lib/kubelet/pods/be2a30d6-be3f-4247-8223-714a89309f07/volumes" Oct 30 14:35:00.118436 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:00.118398 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx"] Oct 30 14:35:00.132546 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:00.132519 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9rqnx"] Oct 30 14:35:00.519035 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:00.518946 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2589b6b-d190-44d1-a68b-329ecd3cc941" path="/var/lib/kubelet/pods/c2589b6b-d190-44d1-a68b-329ecd3cc941/volumes" Oct 30 14:35:20.442505 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:35:20.442465 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:35:28.040913 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:28.040871 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx"] Oct 30 14:35:28.053033 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:28.053001 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-h46nx"] Oct 30 14:35:28.517027 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:28.516993 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20f347e-1957-4233-9a5a-7e8a1e40410d" path="/var/lib/kubelet/pods/d20f347e-1957-4233-9a5a-7e8a1e40410d/volumes" Oct 30 14:35:38.078145 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:38.078106 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d"] Oct 30 14:35:38.090321 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:38.090293 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tg87d"] Oct 30 14:35:38.517273 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:38.517236 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf43098-a448-435a-a798-c156f5c5cde9" path="/var/lib/kubelet/pods/cbf43098-a448-435a-a798-c156f5c5cde9/volumes" Oct 30 14:35:45.979607 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:45.979578 4968 scope.go:122] "RemoveContainer" containerID="ada87fa51b793f8dbacf3fe18333c18239772b45a6d638dffcc1434ab6b52745" Oct 30 14:35:45.989536 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:45.989514 4968 scope.go:122] "RemoveContainer" containerID="d8b18a01d6afcd002627c8f4727c05a0ff2f79e1b7032e19fee60961c97fda95" Oct 30 14:35:50.443321 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:35:50.443281 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:35:58.551967 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.551888 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj"] Oct 30 14:35:58.552431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552412 4968 cpu_manager.go:414] "containerMap: removing container" podUID="be2a30d6-be3f-4247-8223-714a89309f07" containerName="manager" Oct 30 14:35:58.552485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552433 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2a30d6-be3f-4247-8223-714a89309f07" containerName="manager" Oct 30 14:35:58.552485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552455 4968 cpu_manager.go:414] "containerMap: removing container" podUID="5ef5f1d6-ef08-4831-8069-027d0c1f46f8" containerName="payload-processing" Oct 30 14:35:58.552485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552461 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef5f1d6-ef08-4831-8069-027d0c1f46f8" containerName="payload-processing" Oct 30 14:35:58.552485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552476 4968 cpu_manager.go:414] "containerMap: removing container" podUID="202c76db-19bc-4c08-8071-8cff9dda8de6" containerName="revoke-keys" Oct 30 14:35:58.552485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552482 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202c76db-19bc-4c08-8071-8cff9dda8de6" containerName="revoke-keys" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552490 4968 cpu_manager.go:414] "containerMap: removing container" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552496 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552508 4968 cpu_manager.go:414] "containerMap: removing container" podUID="bb0117f0-b7d7-43b1-aa23-b435f22d8465" containerName="payload-pre-processing" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552513 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0117f0-b7d7-43b1-aa23-b435f22d8465" containerName="payload-pre-processing" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552522 4968 cpu_manager.go:414] "containerMap: removing container" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552527 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552586 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef5f1d6-ef08-4831-8069-027d0c1f46f8" containerName="payload-processing" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552596 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="202c76db-19bc-4c08-8071-8cff9dda8de6" containerName="revoke-keys" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552604 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552611 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0117f0-b7d7-43b1-aa23-b435f22d8465" containerName="payload-pre-processing" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552616 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a4b3ae-235b-4e8c-97ba-a26813153e7f" containerName="istio-proxy" Oct 30 14:35:58.552635 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.552622 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2a30d6-be3f-4247-8223-714a89309f07" containerName="manager" Oct 30 14:35:58.557172 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.557157 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" Oct 30 14:35:58.571595 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.571573 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj"] Oct 30 14:35:58.581278 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.581253 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wnlf4\"" Oct 30 14:35:58.581497 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.581473 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed0c76b4-f80b-4b08-b3c2-a77524625e1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlknj\" (UID: \"ed0c76b4-f80b-4b08-b3c2-a77524625e1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" Oct 30 14:35:58.581601 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.581585 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6n6\" (UniqueName: \"kubernetes.io/projected/ed0c76b4-f80b-4b08-b3c2-a77524625e1a-kube-api-access-xs6n6\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlknj\" (UID: \"ed0c76b4-f80b-4b08-b3c2-a77524625e1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" Oct 30 14:35:58.682178 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.682148 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6n6\" (UniqueName: \"kubernetes.io/projected/ed0c76b4-f80b-4b08-b3c2-a77524625e1a-kube-api-access-xs6n6\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlknj\" (UID: \"ed0c76b4-f80b-4b08-b3c2-a77524625e1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" Oct 30 14:35:58.682356 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.682207 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed0c76b4-f80b-4b08-b3c2-a77524625e1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlknj\" (UID: \"ed0c76b4-f80b-4b08-b3c2-a77524625e1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" Oct 30 14:35:58.682521 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.682506 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed0c76b4-f80b-4b08-b3c2-a77524625e1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlknj\" (UID: \"ed0c76b4-f80b-4b08-b3c2-a77524625e1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" Oct 30 14:35:58.708943 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.708919 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6n6\" (UniqueName: \"kubernetes.io/projected/ed0c76b4-f80b-4b08-b3c2-a77524625e1a-kube-api-access-xs6n6\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlknj\" (UID: \"ed0c76b4-f80b-4b08-b3c2-a77524625e1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" Oct 30 14:35:58.870144 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:58.870116 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" Oct 30 14:35:59.208193 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:59.208168 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj"] Oct 30 14:35:59.214155 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:35:59.214124 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0c76b4_f80b_4b08_b3c2_a77524625e1a.slice/crio-9ba275749c9d01d6a2cfc1d04fb3cfc16c53ab5dff2c542f0c237249cd5fa192 WatchSource:0}: Error finding container 9ba275749c9d01d6a2cfc1d04fb3cfc16c53ab5dff2c542f0c237249cd5fa192: Status 404 returned error can't find the container with id 9ba275749c9d01d6a2cfc1d04fb3cfc16c53ab5dff2c542f0c237249cd5fa192 Oct 30 14:35:59.537223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:59.537140 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" event={"ID":"ed0c76b4-f80b-4b08-b3c2-a77524625e1a","Type":"ContainerStarted","Data":"ec5c4e6c334a26a1cdb82acd5f02dc7d4ceb66ee706340e353aa9fc6e9bcc997"} Oct 30 14:35:59.537223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:59.537198 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" event={"ID":"ed0c76b4-f80b-4b08-b3c2-a77524625e1a","Type":"ContainerStarted","Data":"9ba275749c9d01d6a2cfc1d04fb3cfc16c53ab5dff2c542f0c237249cd5fa192"} Oct 30 14:35:59.537410 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:59.537385 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" Oct 30 14:35:59.595874 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:35:59.595821 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" podStartSLOduration=1.595808017 podStartE2EDuration="1.595808017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:59.594026929 +0000 UTC m=+1755.219470301" watchObservedRunningTime="2026-10-30 14:35:59.595808017 +0000 UTC m=+1755.221251390" Oct 30 14:36:00.035732 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:00.035684 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2"] Oct 30 14:36:00.042176 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:00.042148 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gbws2"] Oct 30 14:36:00.517427 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:00.517396 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8123d50-7fb9-4d4e-993c-01d4dc12a002" path="/var/lib/kubelet/pods/c8123d50-7fb9-4d4e-993c-01d4dc12a002/volumes" Oct 30 14:36:10.544365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:10.544332 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlknj" Oct 30 14:36:17.390658 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:17.390626 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:36:20.445427 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:36:20.445389 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:36:21.563168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.563130 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s"] Oct 30 14:36:21.566911 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.566888 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:21.589900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.589864 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:36:21.601985 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.601952 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:36:21.603119 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.603095 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:36:21.603572 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.603531 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:36:21.607722 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.607680 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s"] Oct 30 14:36:21.632815 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.632774 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l7j8s\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:21.632981 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.632892 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x948w\" (UniqueName: \"kubernetes.io/projected/3445f128-0610-4c65-a1b2-fa779653e8ff-kube-api-access-x948w\") pod \"maas-api-revoke-sub-d576e68b-l7j8s\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:21.633044 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.633017 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l7j8s\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:21.734302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.734265 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x948w\" (UniqueName: \"kubernetes.io/projected/3445f128-0610-4c65-a1b2-fa779653e8ff-kube-api-access-x948w\") pod \"maas-api-revoke-sub-d576e68b-l7j8s\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:21.734514 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.734329 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l7j8s\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:21.734514 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.734369 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l7j8s\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:21.735079 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.735051 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l7j8s\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:21.737469 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.737417 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l7j8s\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:21.759783 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.759755 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x948w\" (UniqueName: \"kubernetes.io/projected/3445f128-0610-4c65-a1b2-fa779653e8ff-kube-api-access-x948w\") pod \"maas-api-revoke-sub-d576e68b-l7j8s\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:21.893761 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:21.893733 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:22.035100 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:22.035053 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2"] Oct 30 14:36:22.039092 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:22.039067 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzdv2"] Oct 30 14:36:22.196486 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:22.196460 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s"] Oct 30 14:36:22.198308 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:36:22.198284 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3445f128_0610_4c65_a1b2_fa779653e8ff.slice/crio-4756eca577d5a0eedd89b1558ea581ecd22c44d622760ac6332eddf55dbe7202 WatchSource:0}: Error finding container 4756eca577d5a0eedd89b1558ea581ecd22c44d622760ac6332eddf55dbe7202: Status 404 returned error can't find the container with id 4756eca577d5a0eedd89b1558ea581ecd22c44d622760ac6332eddf55dbe7202 Oct 30 14:36:22.517596 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:22.517570 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9648934-606c-47ba-b8d2-fde42fbfb37b" path="/var/lib/kubelet/pods/b9648934-606c-47ba-b8d2-fde42fbfb37b/volumes" Oct 30 14:36:22.667618 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:22.667581 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" event={"ID":"3445f128-0610-4c65-a1b2-fa779653e8ff","Type":"ContainerStarted","Data":"eebd10a2499d9a1ba086bb0dfa75df0fd57c856708c2e126534829d432d7dce4"} Oct 30 14:36:22.668045 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:22.667639 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" event={"ID":"3445f128-0610-4c65-a1b2-fa779653e8ff","Type":"ContainerStarted","Data":"4756eca577d5a0eedd89b1558ea581ecd22c44d622760ac6332eddf55dbe7202"} Oct 30 14:36:22.708159 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:22.708114 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" podStartSLOduration=1.7081014780000001 podStartE2EDuration="1.708101478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:36:22.707940027 +0000 UTC m=+1778.333383401" watchObservedRunningTime="2026-10-30 14:36:22.708101478 +0000 UTC m=+1778.333544850" Oct 30 14:36:23.674673 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:23.674638 4968 generic.go:356] "Generic (PLEG): container finished" podID="3445f128-0610-4c65-a1b2-fa779653e8ff" containerID="eebd10a2499d9a1ba086bb0dfa75df0fd57c856708c2e126534829d432d7dce4" exitCode=0 Oct 30 14:36:23.675156 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:23.674694 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" event={"ID":"3445f128-0610-4c65-a1b2-fa779653e8ff","Type":"ContainerDied","Data":"eebd10a2499d9a1ba086bb0dfa75df0fd57c856708c2e126534829d432d7dce4"} Oct 30 14:36:24.815867 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:24.815844 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:24.968284 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:24.968211 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-openshift-ca\") pod \"3445f128-0610-4c65-a1b2-fa779653e8ff\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " Oct 30 14:36:24.968284 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:24.968254 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-xks-ca\") pod \"3445f128-0610-4c65-a1b2-fa779653e8ff\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " Oct 30 14:36:24.968461 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:24.968292 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3445f128-0610-4c65-a1b2-fa779653e8ff-kube-api-access-x948w\" (UniqueName: \"kubernetes.io/projected/3445f128-0610-4c65-a1b2-fa779653e8ff-kube-api-access-x948w\") pod \"3445f128-0610-4c65-a1b2-fa779653e8ff\" (UID: \"3445f128-0610-4c65-a1b2-fa779653e8ff\") " Oct 30 14:36:24.968634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:24.968608 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-openshift-ca" pod "3445f128-0610-4c65-a1b2-fa779653e8ff" (UID: "3445f128-0610-4c65-a1b2-fa779653e8ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:36:24.970375 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:24.970347 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-xks-ca" pod "3445f128-0610-4c65-a1b2-fa779653e8ff" (UID: "3445f128-0610-4c65-a1b2-fa779653e8ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:36:24.970375 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:24.970364 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3445f128-0610-4c65-a1b2-fa779653e8ff-kube-api-access-x948w" pod "3445f128-0610-4c65-a1b2-fa779653e8ff" (UID: "3445f128-0610-4c65-a1b2-fa779653e8ff"). InnerVolumeSpecName "kube-api-access-x948w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:36:25.069758 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:25.069708 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:36:25.069758 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:25.069753 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x948w\" (UniqueName: \"kubernetes.io/projected/3445f128-0610-4c65-a1b2-fa779653e8ff-kube-api-access-x948w\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:36:25.069758 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:25.069763 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3445f128-0610-4c65-a1b2-fa779653e8ff-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:36:25.687167 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:25.687136 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" Oct 30 14:36:25.687341 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:25.687168 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s" event={"ID":"3445f128-0610-4c65-a1b2-fa779653e8ff","Type":"ContainerDied","Data":"4756eca577d5a0eedd89b1558ea581ecd22c44d622760ac6332eddf55dbe7202"} Oct 30 14:36:25.687341 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:25.687213 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4756eca577d5a0eedd89b1558ea581ecd22c44d622760ac6332eddf55dbe7202" Oct 30 14:36:32.791946 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:32.791912 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:36:36.570089 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.570044 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq"] Oct 30 14:36:36.570634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.570481 4968 cpu_manager.go:414] "containerMap: removing container" podUID="3445f128-0610-4c65-a1b2-fa779653e8ff" containerName="revoke-keys" Oct 30 14:36:36.570634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.570496 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3445f128-0610-4c65-a1b2-fa779653e8ff" containerName="revoke-keys" Oct 30 14:36:36.570634 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.570571 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="3445f128-0610-4c65-a1b2-fa779653e8ff" containerName="revoke-keys" Oct 30 14:36:36.633075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.633037 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:36.658851 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.650876 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq"] Oct 30 14:36:36.658851 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.657222 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:36:36.668753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.668693 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:36:36.669016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.668758 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:36:36.669180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.668843 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:36:36.739745 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.728225 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ndzkq\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:36.740053 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.740023 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ndzkq\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:36.740268 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.740247 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prcwc\" (UniqueName: \"kubernetes.io/projected/13595076-3edd-4480-a500-5a9d4c626c1e-kube-api-access-prcwc\") pod \"maas-api-revoke-sub-d2bc93bc-ndzkq\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:36.841275 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.841227 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ndzkq\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:36.842146 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.842118 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ndzkq\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:36.843110 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.843091 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prcwc\" (UniqueName: \"kubernetes.io/projected/13595076-3edd-4480-a500-5a9d4c626c1e-kube-api-access-prcwc\") pod \"maas-api-revoke-sub-d2bc93bc-ndzkq\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:36.851990 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.842979 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ndzkq\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:36.864819 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.864780 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ndzkq\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:36.869505 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.869478 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prcwc\" (UniqueName: \"kubernetes.io/projected/13595076-3edd-4480-a500-5a9d4c626c1e-kube-api-access-prcwc\") pod \"maas-api-revoke-sub-d2bc93bc-ndzkq\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:36.974048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:36.974008 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:37.441792 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:37.436409 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq"] Oct 30 14:36:37.463680 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:36:37.463643 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13595076_3edd_4480_a500_5a9d4c626c1e.slice/crio-1a971256efbb72db64eefe77819dd041f02b1f46bb271f33dc5bf40d9dd6fd00 WatchSource:0}: Error finding container 1a971256efbb72db64eefe77819dd041f02b1f46bb271f33dc5bf40d9dd6fd00: Status 404 returned error can't find the container with id 1a971256efbb72db64eefe77819dd041f02b1f46bb271f33dc5bf40d9dd6fd00 Oct 30 14:36:37.800372 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:37.800318 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" event={"ID":"13595076-3edd-4480-a500-5a9d4c626c1e","Type":"ContainerStarted","Data":"1a971256efbb72db64eefe77819dd041f02b1f46bb271f33dc5bf40d9dd6fd00"} Oct 30 14:36:38.807961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:38.807864 4968 generic.go:356] "Generic (PLEG): container finished" podID="13595076-3edd-4480-a500-5a9d4c626c1e" containerID="2d2027f6653dfb529f3460dc0289cde57b45d46df7a114959466a1eb3daf792a" exitCode=0 Oct 30 14:36:38.807961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:38.807947 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" event={"ID":"13595076-3edd-4480-a500-5a9d4c626c1e","Type":"ContainerDied","Data":"2d2027f6653dfb529f3460dc0289cde57b45d46df7a114959466a1eb3daf792a"} Oct 30 14:36:39.952883 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:39.952856 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:40.003458 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.003421 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13595076-3edd-4480-a500-5a9d4c626c1e-kube-api-access-prcwc\" (UniqueName: \"kubernetes.io/projected/13595076-3edd-4480-a500-5a9d4c626c1e-kube-api-access-prcwc\") pod \"13595076-3edd-4480-a500-5a9d4c626c1e\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " Oct 30 14:36:40.003643 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.003519 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-xks-ca\") pod \"13595076-3edd-4480-a500-5a9d4c626c1e\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " Oct 30 14:36:40.003643 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.003623 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-openshift-ca\") pod \"13595076-3edd-4480-a500-5a9d4c626c1e\" (UID: \"13595076-3edd-4480-a500-5a9d4c626c1e\") " Oct 30 14:36:40.004203 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.004177 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-openshift-ca" pod "13595076-3edd-4480-a500-5a9d4c626c1e" (UID: "13595076-3edd-4480-a500-5a9d4c626c1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:36:40.006208 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.006178 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-xks-ca" pod "13595076-3edd-4480-a500-5a9d4c626c1e" (UID: "13595076-3edd-4480-a500-5a9d4c626c1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:36:40.006757 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.006733 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13595076-3edd-4480-a500-5a9d4c626c1e-kube-api-access-prcwc" pod "13595076-3edd-4480-a500-5a9d4c626c1e" (UID: "13595076-3edd-4480-a500-5a9d4c626c1e"). InnerVolumeSpecName "kube-api-access-prcwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:36:40.105108 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.105073 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:36:40.105108 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.105104 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prcwc\" (UniqueName: \"kubernetes.io/projected/13595076-3edd-4480-a500-5a9d4c626c1e-kube-api-access-prcwc\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:36:40.105108 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.105114 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13595076-3edd-4480-a500-5a9d4c626c1e-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:36:40.820022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.819942 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" event={"ID":"13595076-3edd-4480-a500-5a9d4c626c1e","Type":"ContainerDied","Data":"1a971256efbb72db64eefe77819dd041f02b1f46bb271f33dc5bf40d9dd6fd00"} Oct 30 14:36:40.820022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.819983 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq" Oct 30 14:36:40.820022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:40.819993 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a971256efbb72db64eefe77819dd041f02b1f46bb271f33dc5bf40d9dd6fd00" Oct 30 14:36:42.033198 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:42.033165 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5"] Oct 30 14:36:42.045126 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:42.045092 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fzlq5"] Oct 30 14:36:42.517316 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:42.517280 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d95909-1ae3-4b27-a135-c63b9a80d912" path="/var/lib/kubelet/pods/00d95909-1ae3-4b27-a135-c63b9a80d912/volumes" Oct 30 14:36:44.748154 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:44.748117 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:36:44.753585 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:44.753554 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:36:46.030560 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:46.030526 4968 scope.go:122] "RemoveContainer" containerID="d59124fe4e910f313d0cc437fa166a64525e5b74d962b43048558e18c6dd29fc" Oct 30 14:36:46.039454 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:46.039435 4968 scope.go:122] "RemoveContainer" containerID="9be5edd330de55aa84f0c8e3b6cf80a6e995fddf9bcc23f018d3d875ead44190" Oct 30 14:36:46.047534 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:46.047512 4968 scope.go:122] "RemoveContainer" containerID="99ec97a24c30332c17e39f49573bbcd81601de7a4e68836cdad582b8d1f14d81" Oct 30 14:36:48.697668 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.697636 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt"] Oct 30 14:36:48.698168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.698103 4968 cpu_manager.go:414] "containerMap: removing container" podUID="13595076-3edd-4480-a500-5a9d4c626c1e" containerName="revoke-keys" Oct 30 14:36:48.698168 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.698115 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13595076-3edd-4480-a500-5a9d4c626c1e" containerName="revoke-keys" Oct 30 14:36:48.698321 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.698192 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="13595076-3edd-4480-a500-5a9d4c626c1e" containerName="revoke-keys" Oct 30 14:36:48.702632 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.702609 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.718371 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.718353 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-e8c73c-openshift-default-dockercfg-jvj4x\"" Oct 30 14:36:48.732996 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.732966 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt"] Oct 30 14:36:48.781537 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.781500 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-envoy\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.781756 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.781546 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-data\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.781756 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.781702 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-token\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.781898 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.781815 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-credential-socket\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.781898 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.781862 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-socket\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.781898 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.781886 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-certs\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.782090 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.782057 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-podinfo\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.782172 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.782146 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p982t\" (UniqueName: \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-kube-api-access-p982t\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.782218 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.782202 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istiod-ca-cert\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.883384 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.883351 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istiod-ca-cert\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.883592 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.883410 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-envoy\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.883592 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.883445 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-data\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.883592 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.883479 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-token\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.883592 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.883515 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-credential-socket\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.883592 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.883549 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-socket\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.883592 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.883573 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-certs\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.883955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.883677 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-podinfo\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.883955 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.883755 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p982t\" (UniqueName: \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-kube-api-access-p982t\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.884611 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.884586 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-credential-socket\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.884782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.884622 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istiod-ca-cert\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.885009 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.884987 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-socket\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.885527 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.885502 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-certs\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.886111 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.886083 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-data\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.887993 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.887966 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-envoy\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.888309 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.888285 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-podinfo\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.908201 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.908145 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-token\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:48.910783 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:48.910753 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p982t\" (UniqueName: \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-kube-api-access-p982t\") pod \"e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:49.016048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:49.015958 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:49.336577 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:49.336550 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt"] Oct 30 14:36:49.338040 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:36:49.338007 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5f23ef_38ec_4aa4_9d0c_dae217d08b0a.slice/crio-f55c91ea35264eb72790889151e18a56a4413bb662b98f57533b00721d7e7549 WatchSource:0}: Error finding container f55c91ea35264eb72790889151e18a56a4413bb662b98f57533b00721d7e7549: Status 404 returned error can't find the container with id f55c91ea35264eb72790889151e18a56a4413bb662b98f57533b00721d7e7549 Oct 30 14:36:49.340372 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:49.340332 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:36:49.340465 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:49.340404 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:36:49.340465 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:49.340440 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:36:49.880479 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:49.880440 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" event={"ID":"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a","Type":"ContainerStarted","Data":"de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9"} Oct 30 14:36:49.880997 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:49.880492 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" event={"ID":"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a","Type":"ContainerStarted","Data":"f55c91ea35264eb72790889151e18a56a4413bb662b98f57533b00721d7e7549"} Oct 30 14:36:49.939840 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:49.939780 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" podStartSLOduration=1.9397609070000001 podStartE2EDuration="1.939760907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:36:49.93729542 +0000 UTC m=+1805.562738817" watchObservedRunningTime="2026-10-30 14:36:49.939760907 +0000 UTC m=+1805.565204281" Oct 30 14:36:50.016964 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:50.016928 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:50.022201 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:50.022175 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:50.451421 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:36:50.451375 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:36:50.886855 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:50.886821 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:50.887899 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:50.887881 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:36:54.144704 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.144671 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9"] Oct 30 14:36:54.148960 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.148937 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:36:54.166794 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.166766 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e8c73c\"" Oct 30 14:36:54.167049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.167028 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b8915fd0-dockercfg-fv5kg\"" Oct 30 14:36:54.169382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.169357 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9"] Oct 30 14:36:54.267760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.264331 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nrd\" (UniqueName: \"kubernetes.io/projected/80f733f8-6cc4-415a-8162-bd84bb41d91e-kube-api-access-v4nrd\") pod \"payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9\" (UID: \"80f733f8-6cc4-415a-8162-bd84bb41d91e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:36:54.270473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.270426 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f733f8-6cc4-415a-8162-bd84bb41d91e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9\" (UID: \"80f733f8-6cc4-415a-8162-bd84bb41d91e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:36:54.270671 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.270651 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd"] Oct 30 14:36:54.275998 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.275969 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd"] Oct 30 14:36:54.276138 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.276091 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:36:54.371437 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.371404 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f733f8-6cc4-415a-8162-bd84bb41d91e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9\" (UID: \"80f733f8-6cc4-415a-8162-bd84bb41d91e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:36:54.371613 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.371481 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0220c644-0263-4f16-95a3-1745ac69cdfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd\" (UID: \"0220c644-0263-4f16-95a3-1745ac69cdfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:36:54.371613 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.371517 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nrd\" (UniqueName: \"kubernetes.io/projected/80f733f8-6cc4-415a-8162-bd84bb41d91e-kube-api-access-v4nrd\") pod \"payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9\" (UID: \"80f733f8-6cc4-415a-8162-bd84bb41d91e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:36:54.371613 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.371589 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkffm\" (UniqueName: \"kubernetes.io/projected/0220c644-0263-4f16-95a3-1745ac69cdfb-kube-api-access-kkffm\") pod \"payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd\" (UID: \"0220c644-0263-4f16-95a3-1745ac69cdfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:36:54.372122 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.372102 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f733f8-6cc4-415a-8162-bd84bb41d91e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9\" (UID: \"80f733f8-6cc4-415a-8162-bd84bb41d91e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:36:54.400516 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.400453 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nrd\" (UniqueName: \"kubernetes.io/projected/80f733f8-6cc4-415a-8162-bd84bb41d91e-kube-api-access-v4nrd\") pod \"payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9\" (UID: \"80f733f8-6cc4-415a-8162-bd84bb41d91e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:36:54.462975 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.462947 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:36:54.472470 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.472441 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkffm\" (UniqueName: \"kubernetes.io/projected/0220c644-0263-4f16-95a3-1745ac69cdfb-kube-api-access-kkffm\") pod \"payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd\" (UID: \"0220c644-0263-4f16-95a3-1745ac69cdfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:36:54.472587 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.472569 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0220c644-0263-4f16-95a3-1745ac69cdfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd\" (UID: \"0220c644-0263-4f16-95a3-1745ac69cdfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:36:54.473280 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.473257 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0220c644-0263-4f16-95a3-1745ac69cdfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd\" (UID: \"0220c644-0263-4f16-95a3-1745ac69cdfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:36:54.496354 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.496325 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkffm\" (UniqueName: \"kubernetes.io/projected/0220c644-0263-4f16-95a3-1745ac69cdfb-kube-api-access-kkffm\") pod \"payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd\" (UID: \"0220c644-0263-4f16-95a3-1745ac69cdfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:36:54.591140 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.591088 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:36:54.765351 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.765315 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9"] Oct 30 14:36:54.911857 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.911808 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" event={"ID":"80f733f8-6cc4-415a-8162-bd84bb41d91e","Type":"ContainerStarted","Data":"a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067"} Oct 30 14:36:54.912039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.911888 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" event={"ID":"80f733f8-6cc4-415a-8162-bd84bb41d91e","Type":"ContainerStarted","Data":"d9b804d07b2e4323a8c7eaa460fb0e6384b2af63dbce0377d55ca00ac261ff92"} Oct 30 14:36:54.912039 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.912014 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:36:54.920461 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.920435 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd"] Oct 30 14:36:54.921731 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:36:54.921698 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0220c644_0263_4f16_95a3_1745ac69cdfb.slice/crio-6fb968fdc97d14aed6bd37544bc652bcd8af70c8d8612b9fc92444fbe7940efe WatchSource:0}: Error finding container 6fb968fdc97d14aed6bd37544bc652bcd8af70c8d8612b9fc92444fbe7940efe: Status 404 returned error can't find the container with id 6fb968fdc97d14aed6bd37544bc652bcd8af70c8d8612b9fc92444fbe7940efe Oct 30 14:36:54.954979 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:54.954936 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" podStartSLOduration=0.95492415 podStartE2EDuration="954.92415ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:36:54.954251983 +0000 UTC m=+1810.579695357" watchObservedRunningTime="2026-10-30 14:36:54.95492415 +0000 UTC m=+1810.580367522" Oct 30 14:36:55.919470 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:55.919429 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" event={"ID":"0220c644-0263-4f16-95a3-1745ac69cdfb","Type":"ContainerStarted","Data":"e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016"} Oct 30 14:36:55.919993 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:55.919496 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" event={"ID":"0220c644-0263-4f16-95a3-1745ac69cdfb","Type":"ContainerStarted","Data":"6fb968fdc97d14aed6bd37544bc652bcd8af70c8d8612b9fc92444fbe7940efe"} Oct 30 14:36:55.919993 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:55.919632 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:36:55.962661 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:55.962606 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" podStartSLOduration=1.9625915699999998 podStartE2EDuration="1.96259157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:36:55.960872181 +0000 UTC m=+1811.586315567" watchObservedRunningTime="2026-10-30 14:36:55.96259157 +0000 UTC m=+1811.588034943" Oct 30 14:36:57.547747 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:36:57.547689 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:37:03.043175 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:03.043133 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9"] Oct 30 14:37:03.055078 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:03.055047 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bpdj9"] Oct 30 14:37:04.518094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:04.518021 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f45bc9-ac20-449f-a077-99cb1f8fb6c7" path="/var/lib/kubelet/pods/a7f45bc9-ac20-449f-a077-99cb1f8fb6c7/volumes" Oct 30 14:37:05.922973 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:05.922933 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:37:06.928467 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:06.928432 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:37:06.928900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:06.928812 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:37:06.928900 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:06.928856 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:37:07.995325 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:07.995289 4968 generic.go:356] "Generic (PLEG): container finished" podID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerID="de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9" exitCode=0 Oct 30 14:37:07.995795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:07.995364 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" event={"ID":"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a","Type":"ContainerDied","Data":"de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9"} Oct 30 14:37:07.995795 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:07.995770 4968 scope.go:122] "RemoveContainer" containerID="de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9" Oct 30 14:37:07.996948 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:07.996919 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:37:07.997020 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:07.996990 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:37:07.997063 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:07.997022 4968 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:37:09.009182 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:09.009141 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" event={"ID":"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a","Type":"ContainerStarted","Data":"24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d"} Oct 30 14:37:09.016701 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:09.016676 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:37:09.016870 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:09.016848 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 14:37:09.016912 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:09.016892 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 14:37:10.016803 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:10.016767 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 14:37:10.017207 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:10.016827 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 14:37:11.017029 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:11.016997 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 14:37:11.017453 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:11.017049 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 14:37:12.017199 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:12.017151 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 14:37:12.017199 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:12.017202 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 14:37:12.143664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:12.143628 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:37:13.016468 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:13.016431 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 14:37:13.016664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:13.016486 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 14:37:14.016946 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:14.016912 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 14:37:14.017337 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:14.016966 4968 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 14:37:15.020312 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:15.020274 4968 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:37:15.020820 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:15.020630 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:37:15.021300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:15.021277 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:37:20.446522 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:37:20.446482 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:37:26.842834 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:26.842754 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm"] Oct 30 14:37:26.847585 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:26.847567 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:26.864157 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:26.864128 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:37:26.875726 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:26.875684 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:37:26.875872 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:26.875690 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:37:26.875872 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:26.875810 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:37:26.878024 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:26.878000 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm"] Oct 30 14:37:26.889456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:26.889423 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:37:27.004810 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:27.004769 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab42eb27-m4dzm\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:27.004999 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:27.004858 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab42eb27-m4dzm\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:27.004999 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:27.004915 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkk4p\" (UniqueName: \"kubernetes.io/projected/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-kube-api-access-jkk4p\") pod \"maas-api-revoke-sub-ab42eb27-m4dzm\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:27.105611 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:27.105515 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab42eb27-m4dzm\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:27.105611 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:27.105558 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk4p\" (UniqueName: \"kubernetes.io/projected/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-kube-api-access-jkk4p\") pod \"maas-api-revoke-sub-ab42eb27-m4dzm\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:27.105886 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:27.105669 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab42eb27-m4dzm\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:27.106206 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:27.106180 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab42eb27-m4dzm\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:27.108267 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:27.108243 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab42eb27-m4dzm\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:27.129510 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:27.129486 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkk4p\" (UniqueName: \"kubernetes.io/projected/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-kube-api-access-jkk4p\") pod \"maas-api-revoke-sub-ab42eb27-m4dzm\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:27.161321 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:27.161289 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:27.491756 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:27.491710 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm"] Oct 30 14:37:27.495022 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:37:27.494082 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ddf1d0_19b1_46cb_b34c_3d168c81d1d3.slice/crio-838cbdd39bfc0c300ce101cbdc9034cd944622a523fcfbfad0b5a3174acb5a58 WatchSource:0}: Error finding container 838cbdd39bfc0c300ce101cbdc9034cd944622a523fcfbfad0b5a3174acb5a58: Status 404 returned error can't find the container with id 838cbdd39bfc0c300ce101cbdc9034cd944622a523fcfbfad0b5a3174acb5a58 Oct 30 14:37:28.131408 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:28.131366 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" event={"ID":"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3","Type":"ContainerStarted","Data":"cfbbfbd28ead4572c38cf633972ba1064665e66333ac3f79fb47165df9e98027"} Oct 30 14:37:28.131833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:28.131424 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" event={"ID":"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3","Type":"ContainerStarted","Data":"838cbdd39bfc0c300ce101cbdc9034cd944622a523fcfbfad0b5a3174acb5a58"} Oct 30 14:37:28.173308 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:28.173251 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" podStartSLOduration=2.173236439 podStartE2EDuration="2.173236439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:28.172005135 +0000 UTC m=+1843.797448511" watchObservedRunningTime="2026-10-30 14:37:28.173236439 +0000 UTC m=+1843.798679812" Oct 30 14:37:29.142106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:29.142074 4968 generic.go:356] "Generic (PLEG): container finished" podID="39ddf1d0-19b1-46cb-b34c-3d168c81d1d3" containerID="cfbbfbd28ead4572c38cf633972ba1064665e66333ac3f79fb47165df9e98027" exitCode=0 Oct 30 14:37:29.142517 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:29.142147 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" event={"ID":"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3","Type":"ContainerDied","Data":"cfbbfbd28ead4572c38cf633972ba1064665e66333ac3f79fb47165df9e98027"} Oct 30 14:37:30.291693 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:30.291663 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:30.440223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:30.440134 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-kube-api-access-jkk4p\" (UniqueName: \"kubernetes.io/projected/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-kube-api-access-jkk4p\") pod \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " Oct 30 14:37:30.440223 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:30.440179 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-xks-ca\") pod \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " Oct 30 14:37:30.440473 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:30.440226 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-openshift-ca\") pod \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\" (UID: \"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3\") " Oct 30 14:37:30.440739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:30.440689 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-openshift-ca" pod "39ddf1d0-19b1-46cb-b34c-3d168c81d1d3" (UID: "39ddf1d0-19b1-46cb-b34c-3d168c81d1d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:37:30.442286 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:30.442259 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-kube-api-access-jkk4p" pod "39ddf1d0-19b1-46cb-b34c-3d168c81d1d3" (UID: "39ddf1d0-19b1-46cb-b34c-3d168c81d1d3"). InnerVolumeSpecName "kube-api-access-jkk4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:37:30.442375 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:30.442336 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-xks-ca" pod "39ddf1d0-19b1-46cb-b34c-3d168c81d1d3" (UID: "39ddf1d0-19b1-46cb-b34c-3d168c81d1d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:37:30.541069 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:30.541043 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkk4p\" (UniqueName: \"kubernetes.io/projected/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-kube-api-access-jkk4p\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:37:30.541069 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:30.541069 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:37:30.541237 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:30.541079 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:37:31.154526 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:31.154492 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" Oct 30 14:37:31.154750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:31.154533 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm" event={"ID":"39ddf1d0-19b1-46cb-b34c-3d168c81d1d3","Type":"ContainerDied","Data":"838cbdd39bfc0c300ce101cbdc9034cd944622a523fcfbfad0b5a3174acb5a58"} Oct 30 14:37:31.154750 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:31.154598 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="838cbdd39bfc0c300ce101cbdc9034cd944622a523fcfbfad0b5a3174acb5a58" Oct 30 14:37:38.052542 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.052506 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k"] Oct 30 14:37:38.053254 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.053230 4968 cpu_manager.go:414] "containerMap: removing container" podUID="39ddf1d0-19b1-46cb-b34c-3d168c81d1d3" containerName="revoke-keys" Oct 30 14:37:38.053299 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.053257 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ddf1d0-19b1-46cb-b34c-3d168c81d1d3" containerName="revoke-keys" Oct 30 14:37:38.053413 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.053402 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ddf1d0-19b1-46cb-b34c-3d168c81d1d3" containerName="revoke-keys" Oct 30 14:37:38.057126 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.057097 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:38.076608 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.076583 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k"] Oct 30 14:37:38.085442 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.085416 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:37:38.085623 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.085476 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:37:38.085623 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.085489 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:37:38.085841 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.085555 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:37:38.214241 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.214207 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:38.214448 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.214267 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstn5\" (UniqueName: \"kubernetes.io/projected/e25a9608-3fee-46be-a26c-ce69178b4edf-kube-api-access-rstn5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:38.214448 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.214339 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:38.315240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.315151 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:38.315240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.315195 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstn5\" (UniqueName: \"kubernetes.io/projected/e25a9608-3fee-46be-a26c-ce69178b4edf-kube-api-access-rstn5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:38.315240 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.315218 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:38.315932 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.315905 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:38.317659 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.317634 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:38.339532 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.339505 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstn5\" (UniqueName: \"kubernetes.io/projected/e25a9608-3fee-46be-a26c-ce69178b4edf-kube-api-access-rstn5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:38.374843 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.374807 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:38.480001 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.479970 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:37:38.682230 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:38.682196 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k"] Oct 30 14:37:39.201343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:39.201295 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" event={"ID":"e25a9608-3fee-46be-a26c-ce69178b4edf","Type":"ContainerStarted","Data":"f6346478d610dbbecc112f566cd76e420bc185f1ba82714152f659c440e8d63d"} Oct 30 14:37:39.201343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:39.201352 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" event={"ID":"e25a9608-3fee-46be-a26c-ce69178b4edf","Type":"ContainerStarted","Data":"d3d7891c3c33c65c552fe73a63df4acf0b4ab460a28b4834dd2c427db76a2b01"} Oct 30 14:37:39.254033 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:39.253935 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" podStartSLOduration=1.2539205070000001 podStartE2EDuration="1.253920507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:39.241634151 +0000 UTC m=+1854.867077526" watchObservedRunningTime="2026-10-30 14:37:39.253920507 +0000 UTC m=+1854.879363880" Oct 30 14:37:40.208191 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:40.208152 4968 generic.go:356] "Generic (PLEG): container finished" podID="e25a9608-3fee-46be-a26c-ce69178b4edf" containerID="f6346478d610dbbecc112f566cd76e420bc185f1ba82714152f659c440e8d63d" exitCode=0 Oct 30 14:37:40.208691 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:40.208244 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" event={"ID":"e25a9608-3fee-46be-a26c-ce69178b4edf","Type":"ContainerDied","Data":"f6346478d610dbbecc112f566cd76e420bc185f1ba82714152f659c440e8d63d"} Oct 30 14:37:41.351302 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:41.351277 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:41.548359 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:41.548278 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-xks-ca\") pod \"e25a9608-3fee-46be-a26c-ce69178b4edf\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " Oct 30 14:37:41.548359 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:41.548357 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25a9608-3fee-46be-a26c-ce69178b4edf-kube-api-access-rstn5\" (UniqueName: \"kubernetes.io/projected/e25a9608-3fee-46be-a26c-ce69178b4edf-kube-api-access-rstn5\") pod \"e25a9608-3fee-46be-a26c-ce69178b4edf\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " Oct 30 14:37:41.548579 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:41.548460 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-openshift-ca\") pod \"e25a9608-3fee-46be-a26c-ce69178b4edf\" (UID: \"e25a9608-3fee-46be-a26c-ce69178b4edf\") " Oct 30 14:37:41.548895 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:41.548867 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-openshift-ca" pod "e25a9608-3fee-46be-a26c-ce69178b4edf" (UID: "e25a9608-3fee-46be-a26c-ce69178b4edf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:37:41.550485 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:41.550455 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25a9608-3fee-46be-a26c-ce69178b4edf-kube-api-access-rstn5" pod "e25a9608-3fee-46be-a26c-ce69178b4edf" (UID: "e25a9608-3fee-46be-a26c-ce69178b4edf"). InnerVolumeSpecName "kube-api-access-rstn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:37:41.550609 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:41.550512 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-xks-ca" pod "e25a9608-3fee-46be-a26c-ce69178b4edf" (UID: "e25a9608-3fee-46be-a26c-ce69178b4edf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:37:41.649954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:41.649920 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rstn5\" (UniqueName: \"kubernetes.io/projected/e25a9608-3fee-46be-a26c-ce69178b4edf-kube-api-access-rstn5\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:37:41.649954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:41.649950 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:37:41.649954 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:41.649961 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25a9608-3fee-46be-a26c-ce69178b4edf-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:37:42.222294 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:42.222264 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" Oct 30 14:37:42.222478 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:42.222263 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k" event={"ID":"e25a9608-3fee-46be-a26c-ce69178b4edf","Type":"ContainerDied","Data":"d3d7891c3c33c65c552fe73a63df4acf0b4ab460a28b4834dd2c427db76a2b01"} Oct 30 14:37:42.222478 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:42.222390 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3d7891c3c33c65c552fe73a63df4acf0b4ab460a28b4834dd2c427db76a2b01" Oct 30 14:37:46.077890 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:46.077848 4968 scope.go:122] "RemoveContainer" containerID="f35b9553c8d1ce0a71458d735b880223b213f54bf7c82a376de2517a21b7d3dd" Oct 30 14:37:46.087576 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:46.087553 4968 scope.go:122] "RemoveContainer" containerID="45bceb38635bd0a9f4d6aa5c713a3b9424b8892ad665b367c025890320d83330" Oct 30 14:37:46.098604 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:46.098583 4968 scope.go:122] "RemoveContainer" containerID="287d5d675f64bf5bb5d86569db7d7c5aa71c037a2d614d63409806f3657d542a" Oct 30 14:37:48.547524 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:48.547493 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd"] Oct 30 14:37:48.578521 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:48.578490 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9"] Oct 30 14:37:50.445503 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:37:50.445465 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:37:50.480652 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:50.480601 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k"] Oct 30 14:37:50.491558 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:50.491467 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8c73c-9mv6k"] Oct 30 14:37:50.518658 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:50.518614 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e25a9608-3fee-46be-a26c-ce69178b4edf" path="/var/lib/kubelet/pods/e25a9608-3fee-46be-a26c-ce69178b4edf/volumes" Oct 30 14:37:52.339065 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:52.339031 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:37:53.548498 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:53.548456 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" podUID="0220c644-0263-4f16-95a3-1745ac69cdfb" containerName="payload-processing" containerID="cri-o://e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016" gracePeriod=25 Oct 30 14:37:53.579048 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:53.579010 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" podUID="80f733f8-6cc4-415a-8162-bd84bb41d91e" containerName="payload-pre-processing" containerID="cri-o://a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067" gracePeriod=25 Oct 30 14:37:55.921882 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:55.921836 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" podUID="80f733f8-6cc4-415a-8162-bd84bb41d91e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:37:56.927260 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:56.927223 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" podUID="0220c644-0263-4f16-95a3-1745ac69cdfb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:37:57.916022 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:57.915988 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt"] Oct 30 14:37:57.916263 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:37:57.916238 4968 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" containerID="cri-o://24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d" gracePeriod=30 Oct 30 14:38:00.023318 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:00.023275 4968 patch_prober.go:29] interesting pod/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:38:00.023844 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:00.023336 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:38:03.178259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.178233 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:38:03.303015 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.302930 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-token\" (UniqueName: \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-token\") pod \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " Oct 30 14:38:03.303015 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.302998 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-socket\") pod \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " Oct 30 14:38:03.303227 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303045 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-credential-socket\") pod \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " Oct 30 14:38:03.303227 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303093 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-podinfo\") pod \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " Oct 30 14:38:03.303227 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303119 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istiod-ca-cert\") pod \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " Oct 30 14:38:03.303382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303167 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-envoy\") pod \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " Oct 30 14:38:03.303382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303337 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-socket" pod "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" (UID: "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:03.303382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303362 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-data\") pod \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " Oct 30 14:38:03.303536 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303400 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-certs\") pod \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " Oct 30 14:38:03.303536 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303464 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-kube-api-access-p982t\" (UniqueName: \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-kube-api-access-p982t\") pod \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\" (UID: \"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a\") " Oct 30 14:38:03.303644 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303551 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-data" pod "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" (UID: "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:03.303644 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303561 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-credential-socket" pod "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" (UID: "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:03.303790 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303759 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-certs" pod "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" (UID: "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:03.303945 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303921 4968 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:03.303945 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303947 4968 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:03.304096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303963 4968 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:03.304096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303978 4968 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:03.304096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.303986 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istiod-ca-cert" pod "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" (UID: "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:38:03.305620 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.305595 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-kube-api-access-p982t" pod "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" (UID: "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a"). InnerVolumeSpecName "kube-api-access-p982t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:03.305773 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.305753 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-token" pod "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" (UID: "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:03.305833 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.305769 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-podinfo" pod "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" (UID: "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:38:03.305920 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.305906 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-envoy" pod "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" (UID: "bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:03.372765 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.372699 4968 generic.go:356] "Generic (PLEG): container finished" podID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerID="24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d" exitCode=0 Oct 30 14:38:03.372956 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.372775 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" Oct 30 14:38:03.372956 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.372775 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" event={"ID":"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a","Type":"ContainerDied","Data":"24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d"} Oct 30 14:38:03.372956 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.372860 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt" event={"ID":"bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a","Type":"ContainerDied","Data":"f55c91ea35264eb72790889151e18a56a4413bb662b98f57533b00721d7e7549"} Oct 30 14:38:03.372956 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.372885 4968 scope.go:122] "RemoveContainer" containerID="24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d" Oct 30 14:38:03.383046 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.383030 4968 scope.go:122] "RemoveContainer" containerID="de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9" Oct 30 14:38:03.392696 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.392675 4968 scope.go:122] "RemoveContainer" containerID="24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d" Oct 30 14:38:03.393023 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:38:03.393001 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d\": container with ID starting with 24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d not found: ID does not exist" containerID="24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d" Oct 30 14:38:03.393121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.393038 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d"} err="failed to get container status \"24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d\": rpc error: code = NotFound desc = could not find container \"24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d\": container with ID starting with 24a1df0f06327cf947b78064b19f83b9250e3ba1a4742460ca8a37234217034d not found: ID does not exist" Oct 30 14:38:03.393121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.393063 4968 scope.go:122] "RemoveContainer" containerID="de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9" Oct 30 14:38:03.393321 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:38:03.393303 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9\": container with ID starting with de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9 not found: ID does not exist" containerID="de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9" Oct 30 14:38:03.393369 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.393329 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9"} err="failed to get container status \"de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9\": rpc error: code = NotFound desc = could not find container \"de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9\": container with ID starting with de9595370b6ce6954710dfc6cfb82aeadf31359bdc52e02c8e8132ed8c061ab9 not found: ID does not exist" Oct 30 14:38:03.404792 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.404771 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p982t\" (UniqueName: \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-kube-api-access-p982t\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:03.404792 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.404793 4968 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:03.404910 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.404802 4968 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:03.404910 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.404810 4968 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:03.404910 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.404819 4968 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:03.427035 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.427011 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt"] Oct 30 14:38:03.440275 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:03.440246 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e8c73c-openshift-default-d8dbd665c-wvptt"] Oct 30 14:38:04.518778 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:04.518736 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" path="/var/lib/kubelet/pods/bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a/volumes" Oct 30 14:38:04.850080 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:04.850044 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:38:05.921254 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:05.921211 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" podUID="80f733f8-6cc4-415a-8162-bd84bb41d91e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:38:06.940868 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:06.940824 4968 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" podUID="0220c644-0263-4f16-95a3-1745ac69cdfb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:38:13.869313 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:13.869282 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:38:13.872254 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:13.872237 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:38:13.947495 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:13.947462 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0220c644-0263-4f16-95a3-1745ac69cdfb-kube-api-access-kkffm\" (UniqueName: \"kubernetes.io/projected/0220c644-0263-4f16-95a3-1745ac69cdfb-kube-api-access-kkffm\") pod \"0220c644-0263-4f16-95a3-1745ac69cdfb\" (UID: \"0220c644-0263-4f16-95a3-1745ac69cdfb\") " Oct 30 14:38:13.947662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:13.947563 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f733f8-6cc4-415a-8162-bd84bb41d91e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f733f8-6cc4-415a-8162-bd84bb41d91e-plugins-config-volume\") pod \"80f733f8-6cc4-415a-8162-bd84bb41d91e\" (UID: \"80f733f8-6cc4-415a-8162-bd84bb41d91e\") " Oct 30 14:38:13.947662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:13.947586 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f733f8-6cc4-415a-8162-bd84bb41d91e-kube-api-access-v4nrd\" (UniqueName: \"kubernetes.io/projected/80f733f8-6cc4-415a-8162-bd84bb41d91e-kube-api-access-v4nrd\") pod \"80f733f8-6cc4-415a-8162-bd84bb41d91e\" (UID: \"80f733f8-6cc4-415a-8162-bd84bb41d91e\") " Oct 30 14:38:13.947662 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:13.947602 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0220c644-0263-4f16-95a3-1745ac69cdfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0220c644-0263-4f16-95a3-1745ac69cdfb-plugins-config-volume\") pod \"0220c644-0263-4f16-95a3-1745ac69cdfb\" (UID: \"0220c644-0263-4f16-95a3-1745ac69cdfb\") " Oct 30 14:38:13.948043 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:13.948010 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f733f8-6cc4-415a-8162-bd84bb41d91e-plugins-config-volume" pod "80f733f8-6cc4-415a-8162-bd84bb41d91e" (UID: "80f733f8-6cc4-415a-8162-bd84bb41d91e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:38:13.948166 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:13.948043 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0220c644-0263-4f16-95a3-1745ac69cdfb-plugins-config-volume" pod "0220c644-0263-4f16-95a3-1745ac69cdfb" (UID: "0220c644-0263-4f16-95a3-1745ac69cdfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:38:13.949752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:13.949709 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f733f8-6cc4-415a-8162-bd84bb41d91e-kube-api-access-v4nrd" pod "80f733f8-6cc4-415a-8162-bd84bb41d91e" (UID: "80f733f8-6cc4-415a-8162-bd84bb41d91e"). InnerVolumeSpecName "kube-api-access-v4nrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:13.949852 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:13.949755 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0220c644-0263-4f16-95a3-1745ac69cdfb-kube-api-access-kkffm" pod "0220c644-0263-4f16-95a3-1745ac69cdfb" (UID: "0220c644-0263-4f16-95a3-1745ac69cdfb"). InnerVolumeSpecName "kube-api-access-kkffm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:14.049095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.049054 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4nrd\" (UniqueName: \"kubernetes.io/projected/80f733f8-6cc4-415a-8162-bd84bb41d91e-kube-api-access-v4nrd\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:14.049095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.049089 4968 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0220c644-0263-4f16-95a3-1745ac69cdfb-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:14.049095 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.049099 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkffm\" (UniqueName: \"kubernetes.io/projected/0220c644-0263-4f16-95a3-1745ac69cdfb-kube-api-access-kkffm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:14.049339 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.049107 4968 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f733f8-6cc4-415a-8162-bd84bb41d91e-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:14.482283 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.482246 4968 generic.go:356] "Generic (PLEG): container finished" podID="80f733f8-6cc4-415a-8162-bd84bb41d91e" containerID="a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067" exitCode=0 Oct 30 14:38:14.482488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.482292 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" Oct 30 14:38:14.482488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.482330 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" event={"ID":"80f733f8-6cc4-415a-8162-bd84bb41d91e","Type":"ContainerDied","Data":"a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067"} Oct 30 14:38:14.482488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.482390 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9" event={"ID":"80f733f8-6cc4-415a-8162-bd84bb41d91e","Type":"ContainerDied","Data":"d9b804d07b2e4323a8c7eaa460fb0e6384b2af63dbce0377d55ca00ac261ff92"} Oct 30 14:38:14.482488 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.482422 4968 scope.go:122] "RemoveContainer" containerID="a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067" Oct 30 14:38:14.483768 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.483743 4968 generic.go:356] "Generic (PLEG): container finished" podID="0220c644-0263-4f16-95a3-1745ac69cdfb" containerID="e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016" exitCode=0 Oct 30 14:38:14.483873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.483788 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" Oct 30 14:38:14.483873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.483797 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" event={"ID":"0220c644-0263-4f16-95a3-1745ac69cdfb","Type":"ContainerDied","Data":"e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016"} Oct 30 14:38:14.483873 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.483840 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd" event={"ID":"0220c644-0263-4f16-95a3-1745ac69cdfb","Type":"ContainerDied","Data":"6fb968fdc97d14aed6bd37544bc652bcd8af70c8d8612b9fc92444fbe7940efe"} Oct 30 14:38:14.494282 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.494262 4968 scope.go:122] "RemoveContainer" containerID="a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067" Oct 30 14:38:14.494617 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:38:14.494569 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067\": container with ID starting with a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067 not found: ID does not exist" containerID="a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067" Oct 30 14:38:14.495017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.494602 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067"} err="failed to get container status \"a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067\": rpc error: code = NotFound desc = could not find container \"a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067\": container with ID starting with a097207bfb3ce912bc857cd926dda73b9854c3c4006c05886ff76d441cb8e067 not found: ID does not exist" Oct 30 14:38:14.495017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.494628 4968 scope.go:122] "RemoveContainer" containerID="e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016" Oct 30 14:38:14.503222 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.503204 4968 scope.go:122] "RemoveContainer" containerID="e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016" Oct 30 14:38:14.503485 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:38:14.503469 4968 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016\": container with ID starting with e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016 not found: ID does not exist" containerID="e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016" Oct 30 14:38:14.503532 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.503494 4968 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016"} err="failed to get container status \"e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016\": rpc error: code = NotFound desc = could not find container \"e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016\": container with ID starting with e8254075a75821d267f1b2571fc96c673097c97dd349c6cb11e605918e953016 not found: ID does not exist" Oct 30 14:38:14.534493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.534453 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd"] Oct 30 14:38:14.546008 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.545977 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e8c73c-6f69954c98p9qd"] Oct 30 14:38:14.570269 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.570239 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9"] Oct 30 14:38:14.576133 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:14.576104 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e8c73c-8686f6psv9"] Oct 30 14:38:16.517832 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:16.517800 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0220c644-0263-4f16-95a3-1745ac69cdfb" path="/var/lib/kubelet/pods/0220c644-0263-4f16-95a3-1745ac69cdfb/volumes" Oct 30 14:38:16.518215 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:16.518183 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f733f8-6cc4-415a-8162-bd84bb41d91e" path="/var/lib/kubelet/pods/80f733f8-6cc4-415a-8162-bd84bb41d91e/volumes" Oct 30 14:38:20.443938 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:38:20.443902 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:38:21.655490 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:21.655455 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:38:37.054790 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.054754 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:38:37.965753 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.965694 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg"] Oct 30 14:38:37.966374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966348 4968 cpu_manager.go:414] "containerMap: removing container" podUID="80f733f8-6cc4-415a-8162-bd84bb41d91e" containerName="payload-pre-processing" Oct 30 14:38:37.966374 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966374 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f733f8-6cc4-415a-8162-bd84bb41d91e" containerName="payload-pre-processing" Oct 30 14:38:37.966567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966401 4968 cpu_manager.go:414] "containerMap: removing container" podUID="0220c644-0263-4f16-95a3-1745ac69cdfb" containerName="payload-processing" Oct 30 14:38:37.966567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966409 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0220c644-0263-4f16-95a3-1745ac69cdfb" containerName="payload-processing" Oct 30 14:38:37.966567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966425 4968 cpu_manager.go:414] "containerMap: removing container" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" Oct 30 14:38:37.966567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966433 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" Oct 30 14:38:37.966567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966446 4968 cpu_manager.go:414] "containerMap: removing container" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" Oct 30 14:38:37.966567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966453 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" Oct 30 14:38:37.966567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966470 4968 cpu_manager.go:414] "containerMap: removing container" podUID="e25a9608-3fee-46be-a26c-ce69178b4edf" containerName="revoke-keys" Oct 30 14:38:37.966567 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966478 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25a9608-3fee-46be-a26c-ce69178b4edf" containerName="revoke-keys" Oct 30 14:38:37.967059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966601 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f733f8-6cc4-415a-8162-bd84bb41d91e" containerName="payload-pre-processing" Oct 30 14:38:37.967059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966613 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="0220c644-0263-4f16-95a3-1745ac69cdfb" containerName="payload-processing" Oct 30 14:38:37.967059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966623 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" Oct 30 14:38:37.967059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966634 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5f23ef-38ec-4aa4-9d0c-dae217d08b0a" containerName="istio-proxy" Oct 30 14:38:37.967059 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.966646 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25a9608-3fee-46be-a26c-ce69178b4edf" containerName="revoke-keys" Oct 30 14:38:37.970308 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.970287 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:37.994307 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:37.994239 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg"] Oct 30 14:38:38.003761 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.003056 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:38:38.003761 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.003205 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:38:38.003761 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.003388 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:38:38.003761 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.003339 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:38:38.093260 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.093210 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6d8wg\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:38.093821 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.093361 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6d8wg\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:38.093821 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.093505 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd97j\" (UniqueName: \"kubernetes.io/projected/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-kube-api-access-pd97j\") pod \"maas-api-revoke-sub-452b6d7b-6d8wg\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:38.194628 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.194585 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6d8wg\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:38.194849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.194679 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6d8wg\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:38.194849 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.194795 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd97j\" (UniqueName: \"kubernetes.io/projected/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-kube-api-access-pd97j\") pod \"maas-api-revoke-sub-452b6d7b-6d8wg\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:38.195939 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.195902 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6d8wg\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:38.197882 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.197856 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6d8wg\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:38.219667 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.219567 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd97j\" (UniqueName: \"kubernetes.io/projected/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-kube-api-access-pd97j\") pod \"maas-api-revoke-sub-452b6d7b-6d8wg\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:38.289420 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.289383 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:38.622739 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.618459 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg"] Oct 30 14:38:38.627217 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:38:38.627168 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca6bf37_02fa_4e58_9618_bff3c1dbe13c.slice/crio-51e7250793f3a84769ebf35195f3b65f291196bf282f6bc21096348448c40563 WatchSource:0}: Error finding container 51e7250793f3a84769ebf35195f3b65f291196bf282f6bc21096348448c40563: Status 404 returned error can't find the container with id 51e7250793f3a84769ebf35195f3b65f291196bf282f6bc21096348448c40563 Oct 30 14:38:38.640198 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:38.640166 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" event={"ID":"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c","Type":"ContainerStarted","Data":"51e7250793f3a84769ebf35195f3b65f291196bf282f6bc21096348448c40563"} Oct 30 14:38:39.654197 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:39.654155 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" event={"ID":"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c","Type":"ContainerStarted","Data":"6c118e1c7d87e4ec6c0bdd067587d09ff252d3cc7914357aa755bc980b707872"} Oct 30 14:38:40.665215 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:40.665177 4968 generic.go:356] "Generic (PLEG): container finished" podID="6ca6bf37-02fa-4e58-9618-bff3c1dbe13c" containerID="6c118e1c7d87e4ec6c0bdd067587d09ff252d3cc7914357aa755bc980b707872" exitCode=0 Oct 30 14:38:40.665841 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:40.665769 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" event={"ID":"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c","Type":"ContainerDied","Data":"6c118e1c7d87e4ec6c0bdd067587d09ff252d3cc7914357aa755bc980b707872"} Oct 30 14:38:41.807559 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:41.807537 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:41.978896 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:41.978817 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-xks-ca\") pod \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " Oct 30 14:38:41.978896 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:41.978862 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-kube-api-access-pd97j\" (UniqueName: \"kubernetes.io/projected/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-kube-api-access-pd97j\") pod \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " Oct 30 14:38:41.978896 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:41.978894 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-openshift-ca\") pod \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\" (UID: \"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c\") " Oct 30 14:38:41.979295 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:41.979270 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-openshift-ca" pod "6ca6bf37-02fa-4e58-9618-bff3c1dbe13c" (UID: "6ca6bf37-02fa-4e58-9618-bff3c1dbe13c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:38:41.981042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:41.981008 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-xks-ca" pod "6ca6bf37-02fa-4e58-9618-bff3c1dbe13c" (UID: "6ca6bf37-02fa-4e58-9618-bff3c1dbe13c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:38:41.981042 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:41.981022 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-kube-api-access-pd97j" pod "6ca6bf37-02fa-4e58-9618-bff3c1dbe13c" (UID: "6ca6bf37-02fa-4e58-9618-bff3c1dbe13c"). InnerVolumeSpecName "kube-api-access-pd97j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:42.079889 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:42.079844 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:42.079889 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:42.079883 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd97j\" (UniqueName: \"kubernetes.io/projected/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-kube-api-access-pd97j\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:42.079889 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:42.079893 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca6bf37-02fa-4e58-9618-bff3c1dbe13c-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:38:42.677179 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:42.677142 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" Oct 30 14:38:42.677358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:42.677179 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6d8wg" event={"ID":"6ca6bf37-02fa-4e58-9618-bff3c1dbe13c","Type":"ContainerDied","Data":"51e7250793f3a84769ebf35195f3b65f291196bf282f6bc21096348448c40563"} Oct 30 14:38:42.677358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:42.677228 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e7250793f3a84769ebf35195f3b65f291196bf282f6bc21096348448c40563" Oct 30 14:38:46.131000 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:46.130969 4968 scope.go:122] "RemoveContainer" containerID="614aa72b1a816c4f2583fda1567086c819912651444c46f0894cbafdcfa73228" Oct 30 14:38:46.139655 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:46.139632 4968 scope.go:122] "RemoveContainer" containerID="2dfbc420da616a1a1cb54e4a89c5401b9d17725d074737f4a8f519f523552a5c" Oct 30 14:38:46.147450 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:46.147432 4968 scope.go:122] "RemoveContainer" containerID="fcdd31a58982c85d73b9f656bd7cbba3b2eaa9c3e12917bb736783a9b0e1271c" Oct 30 14:38:50.444014 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:38:50.443976 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:38:51.276358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:51.276320 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:38:58.926434 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.926350 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62"] Oct 30 14:38:58.927075 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.927054 4968 cpu_manager.go:414] "containerMap: removing container" podUID="6ca6bf37-02fa-4e58-9618-bff3c1dbe13c" containerName="revoke-keys" Oct 30 14:38:58.927125 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.927079 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca6bf37-02fa-4e58-9618-bff3c1dbe13c" containerName="revoke-keys" Oct 30 14:38:58.927238 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.927225 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca6bf37-02fa-4e58-9618-bff3c1dbe13c" containerName="revoke-keys" Oct 30 14:38:58.932021 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.932003 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:38:58.949388 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.949358 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62"] Oct 30 14:38:58.958131 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.958107 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:38:58.958131 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.958128 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:38:58.958339 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.958140 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:38:58.958597 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.958576 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:38:58.978226 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.978201 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8gm62\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:38:58.978332 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.978241 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8gm62\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:38:58.978428 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:58.978410 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngs5n\" (UniqueName: \"kubernetes.io/projected/3aa38053-f757-417c-b7ea-f74e16cbed9d-kube-api-access-ngs5n\") pod \"maas-api-revoke-sub-3e31a506-8gm62\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:38:59.079592 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:59.079556 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngs5n\" (UniqueName: \"kubernetes.io/projected/3aa38053-f757-417c-b7ea-f74e16cbed9d-kube-api-access-ngs5n\") pod \"maas-api-revoke-sub-3e31a506-8gm62\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:38:59.079791 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:59.079615 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8gm62\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:38:59.079791 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:59.079642 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8gm62\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:38:59.080364 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:59.080344 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8gm62\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:38:59.081983 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:59.081964 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8gm62\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:38:59.103077 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:59.103045 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngs5n\" (UniqueName: \"kubernetes.io/projected/3aa38053-f757-417c-b7ea-f74e16cbed9d-kube-api-access-ngs5n\") pod \"maas-api-revoke-sub-3e31a506-8gm62\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:38:59.245129 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:59.245060 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:38:59.556005 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:59.555976 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62"] Oct 30 14:38:59.557561 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:38:59.557535 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa38053_f757_417c_b7ea_f74e16cbed9d.slice/crio-de41f8b3185be2059ba22ea136221b9693fe8df4231d9c982902a3ebf7a7ec4a WatchSource:0}: Error finding container de41f8b3185be2059ba22ea136221b9693fe8df4231d9c982902a3ebf7a7ec4a: Status 404 returned error can't find the container with id de41f8b3185be2059ba22ea136221b9693fe8df4231d9c982902a3ebf7a7ec4a Oct 30 14:38:59.647341 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:59.647309 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:38:59.872061 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:38:59.872029 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" event={"ID":"3aa38053-f757-417c-b7ea-f74e16cbed9d","Type":"ContainerStarted","Data":"de41f8b3185be2059ba22ea136221b9693fe8df4231d9c982902a3ebf7a7ec4a"} Oct 30 14:39:00.913740 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:00.913617 4968 generic.go:356] "Generic (PLEG): container finished" podID="3aa38053-f757-417c-b7ea-f74e16cbed9d" containerID="989d970dd5e8cc6d79955317f464575fcf50b9c1b12a2fc1246fa1ca530e4611" exitCode=0 Oct 30 14:39:00.914264 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:00.914038 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" event={"ID":"3aa38053-f757-417c-b7ea-f74e16cbed9d","Type":"ContainerDied","Data":"989d970dd5e8cc6d79955317f464575fcf50b9c1b12a2fc1246fa1ca530e4611"} Oct 30 14:39:02.406939 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.406903 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:39:02.470257 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.470221 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-xks-ca\") pod \"3aa38053-f757-417c-b7ea-f74e16cbed9d\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " Oct 30 14:39:02.471051 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.471026 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa38053-f757-417c-b7ea-f74e16cbed9d-kube-api-access-ngs5n\" (UniqueName: \"kubernetes.io/projected/3aa38053-f757-417c-b7ea-f74e16cbed9d-kube-api-access-ngs5n\") pod \"3aa38053-f757-417c-b7ea-f74e16cbed9d\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " Oct 30 14:39:02.471581 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.471563 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-openshift-ca\") pod \"3aa38053-f757-417c-b7ea-f74e16cbed9d\" (UID: \"3aa38053-f757-417c-b7ea-f74e16cbed9d\") " Oct 30 14:39:02.476219 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.472067 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-openshift-ca" pod "3aa38053-f757-417c-b7ea-f74e16cbed9d" (UID: "3aa38053-f757-417c-b7ea-f74e16cbed9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:39:02.476315 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.474269 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa38053-f757-417c-b7ea-f74e16cbed9d-kube-api-access-ngs5n" pod "3aa38053-f757-417c-b7ea-f74e16cbed9d" (UID: "3aa38053-f757-417c-b7ea-f74e16cbed9d"). InnerVolumeSpecName "kube-api-access-ngs5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:39:02.477195 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.477172 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-xks-ca" pod "3aa38053-f757-417c-b7ea-f74e16cbed9d" (UID: "3aa38053-f757-417c-b7ea-f74e16cbed9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:39:02.600181 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.600092 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:39:02.600181 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.600129 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa38053-f757-417c-b7ea-f74e16cbed9d-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:39:02.600181 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.600145 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngs5n\" (UniqueName: \"kubernetes.io/projected/3aa38053-f757-417c-b7ea-f74e16cbed9d-kube-api-access-ngs5n\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:39:02.937472 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.937446 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" Oct 30 14:39:02.937472 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.937461 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8gm62" event={"ID":"3aa38053-f757-417c-b7ea-f74e16cbed9d","Type":"ContainerDied","Data":"de41f8b3185be2059ba22ea136221b9693fe8df4231d9c982902a3ebf7a7ec4a"} Oct 30 14:39:02.937688 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:02.937530 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de41f8b3185be2059ba22ea136221b9693fe8df4231d9c982902a3ebf7a7ec4a" Oct 30 14:39:05.034283 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:05.034243 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf"] Oct 30 14:39:05.046798 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:05.046770 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0fe258-s6cbf"] Oct 30 14:39:06.517232 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:06.517198 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903385c8-a558-448f-aac3-5e7f57d8d05b" path="/var/lib/kubelet/pods/903385c8-a558-448f-aac3-5e7f57d8d05b/volumes" Oct 30 14:39:10.192135 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:10.192090 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:39:14.657020 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:14.656984 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:39:20.443462 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:39:20.443421 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:39:21.457058 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.457030 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw"] Oct 30 14:39:21.457448 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.457435 4968 cpu_manager.go:414] "containerMap: removing container" podUID="3aa38053-f757-417c-b7ea-f74e16cbed9d" containerName="revoke-keys" Oct 30 14:39:21.457497 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.457449 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa38053-f757-417c-b7ea-f74e16cbed9d" containerName="revoke-keys" Oct 30 14:39:21.457548 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.457539 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa38053-f757-417c-b7ea-f74e16cbed9d" containerName="revoke-keys" Oct 30 14:39:21.460767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.460751 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:21.480382 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.480357 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw"] Oct 30 14:39:21.489767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.489704 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:39:21.489767 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.489729 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:39:21.489951 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.489821 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:39:21.489951 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.489885 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:39:21.506015 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.505988 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wk6s\" (UniqueName: \"kubernetes.io/projected/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-kube-api-access-9wk6s\") pod \"maas-api-revoke-sub-7ac5b230-t44jw\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:21.506152 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.506108 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ac5b230-t44jw\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:21.506285 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.506259 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ac5b230-t44jw\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:21.606799 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.606755 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ac5b230-t44jw\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:21.607013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.606834 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wk6s\" (UniqueName: \"kubernetes.io/projected/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-kube-api-access-9wk6s\") pod \"maas-api-revoke-sub-7ac5b230-t44jw\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:21.607013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.606879 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ac5b230-t44jw\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:21.607446 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.607421 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ac5b230-t44jw\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:21.609174 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.609147 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ac5b230-t44jw\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:21.633931 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.633904 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wk6s\" (UniqueName: \"kubernetes.io/projected/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-kube-api-access-9wk6s\") pod \"maas-api-revoke-sub-7ac5b230-t44jw\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:21.772836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:21.772761 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:22.073186 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:22.073154 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw"] Oct 30 14:39:22.075115 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:39:22.075075 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0477e1c6_6a10_4ba7_aaf0_52fbdfc6142a.slice/crio-bb3e8b32a5ed61b7c3c6ff19c75868ef9241f19ddedc0e8e4d37c5056ed2f5fb WatchSource:0}: Error finding container bb3e8b32a5ed61b7c3c6ff19c75868ef9241f19ddedc0e8e4d37c5056ed2f5fb: Status 404 returned error can't find the container with id bb3e8b32a5ed61b7c3c6ff19c75868ef9241f19ddedc0e8e4d37c5056ed2f5fb Oct 30 14:39:22.094194 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:22.094156 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" event={"ID":"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a","Type":"ContainerStarted","Data":"bb3e8b32a5ed61b7c3c6ff19c75868ef9241f19ddedc0e8e4d37c5056ed2f5fb"} Oct 30 14:39:23.100892 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:23.100851 4968 generic.go:356] "Generic (PLEG): container finished" podID="0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a" containerID="472eb4c0cd3fdd7d890c1b8da66980d2b0acce3a49c6c63724fee702a2b1d32b" exitCode=0 Oct 30 14:39:23.101283 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:23.100901 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" event={"ID":"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a","Type":"ContainerDied","Data":"472eb4c0cd3fdd7d890c1b8da66980d2b0acce3a49c6c63724fee702a2b1d32b"} Oct 30 14:39:24.242427 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:24.242402 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:24.333760 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:24.333726 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-xks-ca\") pod \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " Oct 30 14:39:24.333913 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:24.333779 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-kube-api-access-9wk6s\" (UniqueName: \"kubernetes.io/projected/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-kube-api-access-9wk6s\") pod \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " Oct 30 14:39:24.333913 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:24.333905 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-openshift-ca\") pod \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\" (UID: \"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a\") " Oct 30 14:39:24.334270 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:24.334246 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-openshift-ca" pod "0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a" (UID: "0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:39:24.335788 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:24.335761 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-xks-ca" pod "0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a" (UID: "0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:39:24.335914 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:24.335872 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-kube-api-access-9wk6s" pod "0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a" (UID: "0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a"). InnerVolumeSpecName "kube-api-access-9wk6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:39:24.434803 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:24.434724 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:39:24.434803 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:24.434753 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:39:24.434803 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:24.434762 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wk6s\" (UniqueName: \"kubernetes.io/projected/0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a-kube-api-access-9wk6s\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:39:25.111782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:25.111749 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" Oct 30 14:39:25.111782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:25.111774 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac5b230-t44jw" event={"ID":"0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a","Type":"ContainerDied","Data":"bb3e8b32a5ed61b7c3c6ff19c75868ef9241f19ddedc0e8e4d37c5056ed2f5fb"} Oct 30 14:39:25.112016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:25.111837 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb3e8b32a5ed61b7c3c6ff19c75868ef9241f19ddedc0e8e4d37c5056ed2f5fb" Oct 30 14:39:32.849208 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:32.849167 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:39:37.034141 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:37.034094 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587"] Oct 30 14:39:37.045943 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:37.045919 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4w587"] Oct 30 14:39:38.518116 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:38.518079 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202c76db-19bc-4c08-8071-8cff9dda8de6" path="/var/lib/kubelet/pods/202c76db-19bc-4c08-8071-8cff9dda8de6/volumes" Oct 30 14:39:42.457096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:42.457060 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:39:46.185002 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:46.184974 4968 scope.go:122] "RemoveContainer" containerID="be1162a4eb5a3e0b3b24f9b4cd4003063ec56c93fa35b9edb4f7dd985c377c2b" Oct 30 14:39:50.443235 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:39:50.443190 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:39:50.878400 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:50.878369 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:39:57.315193 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.315157 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns"] Oct 30 14:39:57.315844 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.315574 4968 cpu_manager.go:414] "containerMap: removing container" podUID="0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a" containerName="revoke-keys" Oct 30 14:39:57.315844 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.315584 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a" containerName="revoke-keys" Oct 30 14:39:57.315844 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.315668 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a" containerName="revoke-keys" Oct 30 14:39:57.320121 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.320098 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:39:57.339344 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.339318 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns"] Oct 30 14:39:57.348071 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.348050 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:39:57.348299 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.348282 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:39:57.348392 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.348295 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:39:57.348392 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.348311 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:39:57.401017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.400988 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qqsc\" (UniqueName: \"kubernetes.io/projected/5a72b1bd-5931-4afc-bff4-9bed2786a650-kube-api-access-8qqsc\") pod \"maas-api-revoke-sub-f03629db-wm7ns\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:39:57.401180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.401029 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wm7ns\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:39:57.401180 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.401058 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wm7ns\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:39:57.502056 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.502023 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qqsc\" (UniqueName: \"kubernetes.io/projected/5a72b1bd-5931-4afc-bff4-9bed2786a650-kube-api-access-8qqsc\") pod \"maas-api-revoke-sub-f03629db-wm7ns\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:39:57.502248 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.502061 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wm7ns\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:39:57.502248 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.502082 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wm7ns\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:39:57.502752 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.502702 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wm7ns\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:39:57.504422 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.504399 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wm7ns\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:39:57.526807 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.526782 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qqsc\" (UniqueName: \"kubernetes.io/projected/5a72b1bd-5931-4afc-bff4-9bed2786a650-kube-api-access-8qqsc\") pod \"maas-api-revoke-sub-f03629db-wm7ns\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:39:57.632501 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.632472 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:39:57.936243 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:57.936218 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns"] Oct 30 14:39:57.937820 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:39:57.937792 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a72b1bd_5931_4afc_bff4_9bed2786a650.slice/crio-1c65355b740e1dafa7210b5c09825938e5cb79422dba2e2621e9ed8cebda8711 WatchSource:0}: Error finding container 1c65355b740e1dafa7210b5c09825938e5cb79422dba2e2621e9ed8cebda8711: Status 404 returned error can't find the container with id 1c65355b740e1dafa7210b5c09825938e5cb79422dba2e2621e9ed8cebda8711 Oct 30 14:39:58.368319 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:58.368286 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" event={"ID":"5a72b1bd-5931-4afc-bff4-9bed2786a650","Type":"ContainerStarted","Data":"55efbcbf9478f30f6942d01c2b98630b041a21d2e2b871fcda300b69624d612e"} Oct 30 14:39:58.368734 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:58.368341 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" event={"ID":"5a72b1bd-5931-4afc-bff4-9bed2786a650","Type":"ContainerStarted","Data":"1c65355b740e1dafa7210b5c09825938e5cb79422dba2e2621e9ed8cebda8711"} Oct 30 14:39:58.411203 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:58.411155 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" podStartSLOduration=1.4111436849999999 podStartE2EDuration="1.411143685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:58.410074397 +0000 UTC m=+1994.035517773" watchObservedRunningTime="2026-10-30 14:39:58.411143685 +0000 UTC m=+1994.036587084" Oct 30 14:39:59.376304 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:59.376270 4968 generic.go:356] "Generic (PLEG): container finished" podID="5a72b1bd-5931-4afc-bff4-9bed2786a650" containerID="55efbcbf9478f30f6942d01c2b98630b041a21d2e2b871fcda300b69624d612e" exitCode=0 Oct 30 14:39:59.376686 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:39:59.376344 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" event={"ID":"5a72b1bd-5931-4afc-bff4-9bed2786a650","Type":"ContainerDied","Data":"55efbcbf9478f30f6942d01c2b98630b041a21d2e2b871fcda300b69624d612e"} Oct 30 14:40:00.519044 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:00.519021 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:40:00.633376 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:00.633296 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a72b1bd-5931-4afc-bff4-9bed2786a650-kube-api-access-8qqsc\" (UniqueName: \"kubernetes.io/projected/5a72b1bd-5931-4afc-bff4-9bed2786a650-kube-api-access-8qqsc\") pod \"5a72b1bd-5931-4afc-bff4-9bed2786a650\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " Oct 30 14:40:00.633525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:00.633386 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-xks-ca\") pod \"5a72b1bd-5931-4afc-bff4-9bed2786a650\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " Oct 30 14:40:00.633525 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:00.633420 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-openshift-ca\") pod \"5a72b1bd-5931-4afc-bff4-9bed2786a650\" (UID: \"5a72b1bd-5931-4afc-bff4-9bed2786a650\") " Oct 30 14:40:00.633913 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:00.633876 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-openshift-ca" pod "5a72b1bd-5931-4afc-bff4-9bed2786a650" (UID: "5a72b1bd-5931-4afc-bff4-9bed2786a650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:40:00.635481 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:00.635454 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-xks-ca" pod "5a72b1bd-5931-4afc-bff4-9bed2786a650" (UID: "5a72b1bd-5931-4afc-bff4-9bed2786a650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:40:00.635590 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:00.635538 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a72b1bd-5931-4afc-bff4-9bed2786a650-kube-api-access-8qqsc" pod "5a72b1bd-5931-4afc-bff4-9bed2786a650" (UID: "5a72b1bd-5931-4afc-bff4-9bed2786a650"). InnerVolumeSpecName "kube-api-access-8qqsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:40:00.734343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:00.734305 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:40:00.734343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:00.734334 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a72b1bd-5931-4afc-bff4-9bed2786a650-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:40:00.734343 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:00.734349 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qqsc\" (UniqueName: \"kubernetes.io/projected/5a72b1bd-5931-4afc-bff4-9bed2786a650-kube-api-access-8qqsc\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:40:01.388782 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:01.388746 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" event={"ID":"5a72b1bd-5931-4afc-bff4-9bed2786a650","Type":"ContainerDied","Data":"1c65355b740e1dafa7210b5c09825938e5cb79422dba2e2621e9ed8cebda8711"} Oct 30 14:40:01.388935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:01.388789 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wm7ns" Oct 30 14:40:01.388935 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:01.388799 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c65355b740e1dafa7210b5c09825938e5cb79422dba2e2621e9ed8cebda8711" Oct 30 14:40:04.357668 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:04.357624 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:40:10.077771 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.077711 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd"] Oct 30 14:40:10.078169 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.078153 4968 cpu_manager.go:414] "containerMap: removing container" podUID="5a72b1bd-5931-4afc-bff4-9bed2786a650" containerName="revoke-keys" Oct 30 14:40:10.078169 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.078166 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a72b1bd-5931-4afc-bff4-9bed2786a650" containerName="revoke-keys" Oct 30 14:40:10.078259 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.078248 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a72b1bd-5931-4afc-bff4-9bed2786a650" containerName="revoke-keys" Oct 30 14:40:10.081318 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.081299 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:10.097196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.097173 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:40:10.108087 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.108062 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:40:10.110599 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.110576 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd"] Oct 30 14:40:10.125137 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.125111 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:40:10.125264 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.125200 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:40:10.158875 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.158850 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-58zxd\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:10.159014 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.158885 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7w7\" (UniqueName: \"kubernetes.io/projected/5225360c-9721-452b-b72d-c96e5d1f7e32-kube-api-access-rg7w7\") pod \"maas-api-revoke-sub-0165d7d2-58zxd\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:10.159014 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.159001 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-58zxd\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:10.259854 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.259819 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-58zxd\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:10.260013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.259864 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7w7\" (UniqueName: \"kubernetes.io/projected/5225360c-9721-452b-b72d-c96e5d1f7e32-kube-api-access-rg7w7\") pod \"maas-api-revoke-sub-0165d7d2-58zxd\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:10.260013 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.259902 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-58zxd\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:10.260534 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.260516 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-58zxd\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:10.262221 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.262198 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-58zxd\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:10.284187 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.284160 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7w7\" (UniqueName: \"kubernetes.io/projected/5225360c-9721-452b-b72d-c96e5d1f7e32-kube-api-access-rg7w7\") pod \"maas-api-revoke-sub-0165d7d2-58zxd\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:10.394177 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.394155 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:10.693462 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:10.693437 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd"] Oct 30 14:40:10.695376 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:40:10.695353 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5225360c_9721_452b_b72d_c96e5d1f7e32.slice/crio-e0c1f3f04e1c6232dd5b1f70dfaf874d63b0f23671518b84db2c4286deee656a WatchSource:0}: Error finding container e0c1f3f04e1c6232dd5b1f70dfaf874d63b0f23671518b84db2c4286deee656a: Status 404 returned error can't find the container with id e0c1f3f04e1c6232dd5b1f70dfaf874d63b0f23671518b84db2c4286deee656a Oct 30 14:40:11.469523 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:11.469486 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" event={"ID":"5225360c-9721-452b-b72d-c96e5d1f7e32","Type":"ContainerStarted","Data":"8fdd18ae978d8079d4c5eeecb81051b4ed0190c1a8c1d311dbf30283b2d48f30"} Oct 30 14:40:11.469963 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:11.469566 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" event={"ID":"5225360c-9721-452b-b72d-c96e5d1f7e32","Type":"ContainerStarted","Data":"e0c1f3f04e1c6232dd5b1f70dfaf874d63b0f23671518b84db2c4286deee656a"} Oct 30 14:40:11.512094 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:11.512049 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" podStartSLOduration=1.5120354470000001 podStartE2EDuration="1.512035447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:11.510025388 +0000 UTC m=+2007.135468753" watchObservedRunningTime="2026-10-30 14:40:11.512035447 +0000 UTC m=+2007.137478816" Oct 30 14:40:12.476942 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:12.476904 4968 generic.go:356] "Generic (PLEG): container finished" podID="5225360c-9721-452b-b72d-c96e5d1f7e32" containerID="8fdd18ae978d8079d4c5eeecb81051b4ed0190c1a8c1d311dbf30283b2d48f30" exitCode=0 Oct 30 14:40:12.477337 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:12.476993 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" event={"ID":"5225360c-9721-452b-b72d-c96e5d1f7e32","Type":"ContainerDied","Data":"8fdd18ae978d8079d4c5eeecb81051b4ed0190c1a8c1d311dbf30283b2d48f30"} Oct 30 14:40:13.615746 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:13.615703 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:13.690565 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:13.690529 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-openshift-ca\") pod \"5225360c-9721-452b-b72d-c96e5d1f7e32\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " Oct 30 14:40:13.690764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:13.690616 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-xks-ca\") pod \"5225360c-9721-452b-b72d-c96e5d1f7e32\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " Oct 30 14:40:13.690764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:13.690699 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5225360c-9721-452b-b72d-c96e5d1f7e32-kube-api-access-rg7w7\" (UniqueName: \"kubernetes.io/projected/5225360c-9721-452b-b72d-c96e5d1f7e32-kube-api-access-rg7w7\") pod \"5225360c-9721-452b-b72d-c96e5d1f7e32\" (UID: \"5225360c-9721-452b-b72d-c96e5d1f7e32\") " Oct 30 14:40:13.690989 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:13.690959 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-openshift-ca" pod "5225360c-9721-452b-b72d-c96e5d1f7e32" (UID: "5225360c-9721-452b-b72d-c96e5d1f7e32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:40:13.692916 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:13.692889 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5225360c-9721-452b-b72d-c96e5d1f7e32-kube-api-access-rg7w7" pod "5225360c-9721-452b-b72d-c96e5d1f7e32" (UID: "5225360c-9721-452b-b72d-c96e5d1f7e32"). InnerVolumeSpecName "kube-api-access-rg7w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:40:13.693106 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:13.693078 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-xks-ca" pod "5225360c-9721-452b-b72d-c96e5d1f7e32" (UID: "5225360c-9721-452b-b72d-c96e5d1f7e32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:40:13.792341 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:13.792266 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:40:13.792341 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:13.792293 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5225360c-9721-452b-b72d-c96e5d1f7e32-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:40:13.792341 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:13.792303 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg7w7\" (UniqueName: \"kubernetes.io/projected/5225360c-9721-452b-b72d-c96e5d1f7e32-kube-api-access-rg7w7\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:40:14.489067 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:14.489033 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" event={"ID":"5225360c-9721-452b-b72d-c96e5d1f7e32","Type":"ContainerDied","Data":"e0c1f3f04e1c6232dd5b1f70dfaf874d63b0f23671518b84db2c4286deee656a"} Oct 30 14:40:14.489067 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:14.489064 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-58zxd" Oct 30 14:40:14.489268 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:14.489084 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c1f3f04e1c6232dd5b1f70dfaf874d63b0f23671518b84db2c4286deee656a" Oct 30 14:40:20.443489 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:40:20.443451 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:40:20.980572 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:20.980536 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9"] Oct 30 14:40:20.981082 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:20.981062 4968 cpu_manager.go:414] "containerMap: removing container" podUID="5225360c-9721-452b-b72d-c96e5d1f7e32" containerName="revoke-keys" Oct 30 14:40:20.981202 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:20.981083 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5225360c-9721-452b-b72d-c96e5d1f7e32" containerName="revoke-keys" Oct 30 14:40:20.981202 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:20.981186 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="5225360c-9721-452b-b72d-c96e5d1f7e32" containerName="revoke-keys" Oct 30 14:40:20.984431 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:20.984402 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:21.004896 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.004866 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9"] Oct 30 14:40:21.013688 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.013648 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:40:21.013876 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.013858 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4swck\"" Oct 30 14:40:21.021165 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.021138 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:40:21.021300 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.021283 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:40:21.060848 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.060820 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-4hbw9\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:21.061015 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.060955 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcl5m\" (UniqueName: \"kubernetes.io/projected/7ef30dba-0b58-4d77-b167-50017835468c-kube-api-access-hcl5m\") pod \"maas-api-revoke-sub-6b19338f-4hbw9\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:21.061015 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.061007 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-4hbw9\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:21.162412 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.162364 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-4hbw9\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:21.162578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.162457 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcl5m\" (UniqueName: \"kubernetes.io/projected/7ef30dba-0b58-4d77-b167-50017835468c-kube-api-access-hcl5m\") pod \"maas-api-revoke-sub-6b19338f-4hbw9\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:21.162578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.162515 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-4hbw9\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:21.163196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.163173 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-4hbw9\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:21.164828 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.164810 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-4hbw9\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:21.186991 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.186968 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcl5m\" (UniqueName: \"kubernetes.io/projected/7ef30dba-0b58-4d77-b167-50017835468c-kube-api-access-hcl5m\") pod \"maas-api-revoke-sub-6b19338f-4hbw9\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:21.297925 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.297839 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:21.605678 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:21.605650 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9"] Oct 30 14:40:21.607654 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:40:21.607625 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef30dba_0b58_4d77_b167_50017835468c.slice/crio-75cbaa79807a1f7ff01accadf9ac998c56759b656da068e46cfc172f24beb099 WatchSource:0}: Error finding container 75cbaa79807a1f7ff01accadf9ac998c56759b656da068e46cfc172f24beb099: Status 404 returned error can't find the container with id 75cbaa79807a1f7ff01accadf9ac998c56759b656da068e46cfc172f24beb099 Oct 30 14:40:22.531917 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:22.531880 4968 generic.go:356] "Generic (PLEG): container finished" podID="7ef30dba-0b58-4d77-b167-50017835468c" containerID="da2acaacfcc2a506e80b95aaae2c28c63c1722419386bac774496491ad5dff0e" exitCode=0 Oct 30 14:40:22.532084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:22.531962 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" event={"ID":"7ef30dba-0b58-4d77-b167-50017835468c","Type":"ContainerDied","Data":"da2acaacfcc2a506e80b95aaae2c28c63c1722419386bac774496491ad5dff0e"} Oct 30 14:40:22.532084 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:22.532012 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" event={"ID":"7ef30dba-0b58-4d77-b167-50017835468c","Type":"ContainerStarted","Data":"75cbaa79807a1f7ff01accadf9ac998c56759b656da068e46cfc172f24beb099"} Oct 30 14:40:23.678823 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:23.678797 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:23.789411 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:23.789334 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef30dba-0b58-4d77-b167-50017835468c-kube-api-access-hcl5m\" (UniqueName: \"kubernetes.io/projected/7ef30dba-0b58-4d77-b167-50017835468c-kube-api-access-hcl5m\") pod \"7ef30dba-0b58-4d77-b167-50017835468c\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " Oct 30 14:40:23.789561 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:23.789518 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-openshift-ca\") pod \"7ef30dba-0b58-4d77-b167-50017835468c\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " Oct 30 14:40:23.789561 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:23.789546 4968 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-xks-ca\") pod \"7ef30dba-0b58-4d77-b167-50017835468c\" (UID: \"7ef30dba-0b58-4d77-b167-50017835468c\") " Oct 30 14:40:23.789851 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:23.789824 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-openshift-ca" pod "7ef30dba-0b58-4d77-b167-50017835468c" (UID: "7ef30dba-0b58-4d77-b167-50017835468c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:40:23.791460 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:23.791425 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef30dba-0b58-4d77-b167-50017835468c-kube-api-access-hcl5m" pod "7ef30dba-0b58-4d77-b167-50017835468c" (UID: "7ef30dba-0b58-4d77-b167-50017835468c"). InnerVolumeSpecName "kube-api-access-hcl5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:40:23.791561 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:23.791484 4968 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-xks-ca" pod "7ef30dba-0b58-4d77-b167-50017835468c" (UID: "7ef30dba-0b58-4d77-b167-50017835468c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:40:23.890426 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:23.890400 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-openshift-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:40:23.890426 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:23.890422 4968 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef30dba-0b58-4d77-b167-50017835468c-maas-api-xks-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:40:23.890426 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:23.890433 4968 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcl5m\" (UniqueName: \"kubernetes.io/projected/7ef30dba-0b58-4d77-b167-50017835468c-kube-api-access-hcl5m\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\"" Oct 30 14:40:24.544188 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:24.544161 4968 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" Oct 30 14:40:24.544188 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:24.544178 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-4hbw9" event={"ID":"7ef30dba-0b58-4d77-b167-50017835468c","Type":"ContainerDied","Data":"75cbaa79807a1f7ff01accadf9ac998c56759b656da068e46cfc172f24beb099"} Oct 30 14:40:24.544365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:24.544221 4968 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75cbaa79807a1f7ff01accadf9ac998c56759b656da068e46cfc172f24beb099" Oct 30 14:40:32.274617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:32.274522 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-svfmk"] Oct 30 14:40:43.292020 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:43.291990 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-cjg7m_54b9a0e6-1899-40b6-836a-4dae7752ce36/manager/0.log" Oct 30 14:40:43.826578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:43.826545 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67c6566688-zbvbc_d3f1d457-085f-4d3f-b3fb-f64167c5ac3c/manager/0.log" Oct 30 14:40:44.748569 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:44.748538 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6749b785b4-84kgr_a7bf92ce-2f0a-474c-b69a-a568080a65aa/copy-manifests/0.log" Oct 30 14:40:44.782188 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:44.782165 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6749b785b4-84kgr_a7bf92ce-2f0a-474c-b69a-a568080a65aa/manager/0.log" Oct 30 14:40:46.207130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:46.207098 4968 scope.go:122] "RemoveContainer" containerID="7e77768f0671289a0f00523d473d524d741e7db903dad6816c63563659f8b238" Oct 30 14:40:46.216252 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:46.216235 4968 scope.go:122] "RemoveContainer" containerID="472aae7f9dcd4bbca37cbae666cb75db3415b74fb3df2b972232ba334e1b8c48" Oct 30 14:40:46.224130 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:46.224112 4968 scope.go:122] "RemoveContainer" containerID="cefe0d43f73523cc8f4d09223a045cd66e112fd04cb079fc0b6043a4fa1c90ed" Oct 30 14:40:46.676876 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:46.676846 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-58zxd_5225360c-9721-452b-b72d-c96e5d1f7e32/revoke-keys/0.log" Oct 30 14:40:46.869000 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:46.868978 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-8gm62_3aa38053-f757-417c-b7ea-f74e16cbed9d/revoke-keys/0.log" Oct 30 14:40:47.058205 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:47.058133 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-6d8wg_6ca6bf37-02fa-4e58-9618-bff3c1dbe13c/revoke-keys/0.log" Oct 30 14:40:47.231642 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:47.231610 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-4hbw9_7ef30dba-0b58-4d77-b167-50017835468c/revoke-keys/0.log" Oct 30 14:40:47.422080 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:47.422050 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7ac5b230-t44jw_0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a/revoke-keys/0.log" Oct 30 14:40:47.605797 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:47.605769 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ab42eb27-m4dzm_39ddf1d0-19b1-46cb-b34c-3d168c81d1d3/revoke-keys/0.log" Oct 30 14:40:47.797451 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:47.797377 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-ndzkq_13595076-3edd-4480-a500-5a9d4c626c1e/revoke-keys/0.log" Oct 30 14:40:47.982826 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:47.982798 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-l7j8s_3445f128-0610-4c65-a1b2-fa779653e8ff/revoke-keys/0.log" Oct 30 14:40:48.175779 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:48.175753 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-wm7ns_5a72b1bd-5931-4afc-bff4-9bed2786a650/revoke-keys/0.log" Oct 30 14:40:50.063466 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:50.063432 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-cmbdv_64121261-495b-4cfc-855a-edf6f69d62d0/manager/0.log" Oct 30 14:40:50.285945 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:50.285915 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-x8hl6_ad114249-10f3-4069-b4ee-86543a6e1c46/manager/0.log" Oct 30 14:40:50.443269 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:40:50.443233 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:40:50.691391 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:50.691362 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-mlknj_ed0c76b4-f80b-4b08-b3c2-a77524625e1a/manager/0.log" Oct 30 14:40:50.963597 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:50.963570 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-svfmk_e8e6e17d-44a6-4e4e-bba4-236c02d5a856/limitador/0.log" Oct 30 14:40:52.119685 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:52.119658 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2cgmk_13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b/discovery/0.log" Oct 30 14:40:52.324640 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:52.324613 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-847b5f84b8-7m6sx_9f369d28-d4eb-4d7c-81b6-b80cd88edb80/kube-auth-proxy/0.log" Oct 30 14:40:54.102166 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:54.102135 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-z2bwp_43dc1f23-75ac-490a-9594-6ba78cb3d01e/istio-proxy/0.log" Oct 30 14:40:54.308017 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:54.307991 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-zl46w_3d721901-63ac-4136-a704-ae8283fce2bb/istio-proxy/0.log" Oct 30 14:40:54.501836 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:54.501754 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-zv6j4_ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b/istio-proxy/0.log" Oct 30 14:40:54.855664 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:54.855634 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-j8ntq_76bebba1-752e-47cc-b918-89c30bd62cf5/payload-processing/0.log" Oct 30 14:40:55.047711 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:55.047680 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-598ddffcff-nctk8_dfc6f25a-6a7a-472c-af78-683613ffb7a5/router/0.log" Oct 30 14:40:55.997049 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:55.997020 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg_92142646-e68c-4054-b7bc-7fa5ac83b447/main/0.log" Oct 30 14:40:56.588577 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:56.588549 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-zlt52_bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8/main/0.log" Oct 30 14:40:56.776096 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:40:56.776070 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb_5f2dd677-fdd5-452e-b283-740d207e3890/main/0.log" Oct 30 14:41:07.447311 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:07.447280 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-cjg7m_54b9a0e6-1899-40b6-836a-4dae7752ce36/manager/0.log" Oct 30 14:41:07.945556 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:07.945523 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67c6566688-zbvbc_d3f1d457-085f-4d3f-b3fb-f64167c5ac3c/manager/0.log" Oct 30 14:41:08.821000 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:08.820968 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6749b785b4-84kgr_a7bf92ce-2f0a-474c-b69a-a568080a65aa/copy-manifests/0.log" Oct 30 14:41:08.855638 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:08.855615 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6749b785b4-84kgr_a7bf92ce-2f0a-474c-b69a-a568080a65aa/manager/0.log" Oct 30 14:41:10.466961 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:10.466930 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-58zxd_5225360c-9721-452b-b72d-c96e5d1f7e32/revoke-keys/0.log" Oct 30 14:41:10.666944 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:10.666918 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-8gm62_3aa38053-f757-417c-b7ea-f74e16cbed9d/revoke-keys/0.log" Oct 30 14:41:10.848043 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:10.848012 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-6d8wg_6ca6bf37-02fa-4e58-9618-bff3c1dbe13c/revoke-keys/0.log" Oct 30 14:41:11.042764 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:11.042736 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-4hbw9_7ef30dba-0b58-4d77-b167-50017835468c/revoke-keys/0.log" Oct 30 14:41:11.268195 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:11.268123 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7ac5b230-t44jw_0477e1c6-6a10-4ba7-aaf0-52fbdfc6142a/revoke-keys/0.log" Oct 30 14:41:11.460586 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:11.460558 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ab42eb27-m4dzm_39ddf1d0-19b1-46cb-b34c-3d168c81d1d3/revoke-keys/0.log" Oct 30 14:41:11.688827 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:11.688790 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-ndzkq_13595076-3edd-4480-a500-5a9d4c626c1e/revoke-keys/0.log" Oct 30 14:41:11.885779 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:11.885748 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-l7j8s_3445f128-0610-4c65-a1b2-fa779653e8ff/revoke-keys/0.log" Oct 30 14:41:12.139011 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:12.138984 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-wm7ns_5a72b1bd-5931-4afc-bff4-9bed2786a650/revoke-keys/0.log" Oct 30 14:41:13.637365 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:13.637322 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-cmbdv_64121261-495b-4cfc-855a-edf6f69d62d0/manager/0.log" Oct 30 14:41:13.840405 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:13.840370 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-x8hl6_ad114249-10f3-4069-b4ee-86543a6e1c46/manager/0.log" Oct 30 14:41:14.233468 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:14.233434 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-mlknj_ed0c76b4-f80b-4b08-b3c2-a77524625e1a/manager/0.log" Oct 30 14:41:14.479979 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:14.479954 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-svfmk_e8e6e17d-44a6-4e4e-bba4-236c02d5a856/limitador/0.log" Oct 30 14:41:15.443405 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:15.443377 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2cgmk_13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b/discovery/0.log" Oct 30 14:41:15.656622 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:15.656585 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-847b5f84b8-7m6sx_9f369d28-d4eb-4d7c-81b6-b80cd88edb80/kube-auth-proxy/0.log" Oct 30 14:41:17.498032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:17.498001 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-z2bwp_43dc1f23-75ac-490a-9594-6ba78cb3d01e/istio-proxy/0.log" Oct 30 14:41:17.722253 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:17.722219 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-zl46w_3d721901-63ac-4136-a704-ae8283fce2bb/istio-proxy/0.log" Oct 30 14:41:17.930563 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:17.930532 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-zv6j4_ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b/istio-proxy/0.log" Oct 30 14:41:18.298493 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:18.298411 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-j8ntq_76bebba1-752e-47cc-b918-89c30bd62cf5/payload-processing/0.log" Oct 30 14:41:18.488489 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:18.488460 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-598ddffcff-nctk8_dfc6f25a-6a7a-472c-af78-683613ffb7a5/router/0.log" Oct 30 14:41:19.019355 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:19.019327 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-jrfzg_92142646-e68c-4054-b7bc-7fa5ac83b447/main/0.log" Oct 30 14:41:19.650980 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:19.650952 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-zlt52_bdbd0e4e-7de0-4a42-8cd4-c13fe2e90ae8/main/0.log" Oct 30 14:41:19.854601 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:19.854569 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87hkxb_5f2dd677-fdd5-452e-b283-740d207e3890/main/0.log" Oct 30 14:41:20.443097 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:41:20.443063 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:41:25.055256 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:25.055223 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s"] Oct 30 14:41:25.067606 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:25.067578 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7j8s"] Oct 30 14:41:26.517695 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:26.517660 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3445f128-0610-4c65-a1b2-fa779653e8ff" path="/var/lib/kubelet/pods/3445f128-0610-4c65-a1b2-fa779653e8ff/volumes" Oct 30 14:41:41.046557 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:41.046520 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq"] Oct 30 14:41:41.058027 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:41.057997 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ndzkq"] Oct 30 14:41:42.518163 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:42.518130 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13595076-3edd-4480-a500-5a9d4c626c1e" path="/var/lib/kubelet/pods/13595076-3edd-4480-a500-5a9d4c626c1e/volumes" Oct 30 14:41:44.782196 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:44.782170 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:41:44.788001 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:44.787979 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:41:50.443413 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:41:50.443376 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:41:56.879968 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:56.879887 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67c6566688-zbvbc_d3f1d457-085f-4d3f-b3fb-f64167c5ac3c/manager/0.log" Oct 30 14:41:57.188854 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:57.188770 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67c6566688-zbvbc_d3f1d457-085f-4d3f-b3fb-f64167c5ac3c/manager/0.log" Oct 30 14:41:57.913681 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:57.913655 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-j8ntq_76bebba1-752e-47cc-b918-89c30bd62cf5/payload-processing/0.log" Oct 30 14:41:58.105706 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:41:58.105680 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-j8ntq_76bebba1-752e-47cc-b918-89c30bd62cf5/payload-processing/0.log" Oct 30 14:42:20.443640 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:42:20.443596 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:42:31.038330 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:31.038290 4968 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm"] Oct 30 14:42:31.048379 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:31.048352 4968 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab42eb27-m4dzm"] Oct 30 14:42:32.517653 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:32.517618 4968 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ddf1d0-19b1-46cb-b34c-3d168c81d1d3" path="/var/lib/kubelet/pods/39ddf1d0-19b1-46cb-b34c-3d168c81d1d3/volumes" Oct 30 14:42:43.435183 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:43.435145 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67c6566688-zbvbc_d3f1d457-085f-4d3f-b3fb-f64167c5ac3c/manager/0.log" Oct 30 14:42:43.762448 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:43.762366 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67c6566688-zbvbc_d3f1d457-085f-4d3f-b3fb-f64167c5ac3c/manager/0.log" Oct 30 14:42:44.480759 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:44.480729 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-j8ntq_76bebba1-752e-47cc-b918-89c30bd62cf5/payload-processing/0.log" Oct 30 14:42:44.658599 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:44.658568 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-j8ntq_76bebba1-752e-47cc-b918-89c30bd62cf5/payload-processing/0.log" Oct 30 14:42:45.011098 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.011067 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qw6xv/must-gather-7nbzj"] Oct 30 14:42:45.011533 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.011519 4968 cpu_manager.go:414] "containerMap: removing container" podUID="7ef30dba-0b58-4d77-b167-50017835468c" containerName="revoke-keys" Oct 30 14:42:45.011578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.011535 4968 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef30dba-0b58-4d77-b167-50017835468c" containerName="revoke-keys" Oct 30 14:42:45.011617 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.011599 4968 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef30dba-0b58-4d77-b167-50017835468c" containerName="revoke-keys" Oct 30 14:42:45.014832 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.014814 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qw6xv/must-gather-7nbzj" Oct 30 14:42:45.032373 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.032345 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qw6xv\"/\"openshift-service-ca.crt\"" Oct 30 14:42:45.032474 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.032390 4968 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qw6xv\"/\"kube-root-ca.crt\"" Oct 30 14:42:45.044261 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.044246 4968 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qw6xv\"/\"default-dockercfg-vfvlx\"" Oct 30 14:42:45.051892 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.051869 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qw6xv/must-gather-7nbzj"] Oct 30 14:42:45.128605 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.128569 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9bb854df-b105-409f-9769-081d65756bd0-must-gather-output\") pod \"must-gather-7nbzj\" (UID: \"9bb854df-b105-409f-9769-081d65756bd0\") " pod="openshift-must-gather-qw6xv/must-gather-7nbzj" Oct 30 14:42:45.128806 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.128612 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwwr\" (UniqueName: \"kubernetes.io/projected/9bb854df-b105-409f-9769-081d65756bd0-kube-api-access-5hwwr\") pod \"must-gather-7nbzj\" (UID: \"9bb854df-b105-409f-9769-081d65756bd0\") " pod="openshift-must-gather-qw6xv/must-gather-7nbzj" Oct 30 14:42:45.229823 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.229786 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9bb854df-b105-409f-9769-081d65756bd0-must-gather-output\") pod \"must-gather-7nbzj\" (UID: \"9bb854df-b105-409f-9769-081d65756bd0\") " pod="openshift-must-gather-qw6xv/must-gather-7nbzj" Oct 30 14:42:45.230032 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.229832 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwwr\" (UniqueName: \"kubernetes.io/projected/9bb854df-b105-409f-9769-081d65756bd0-kube-api-access-5hwwr\") pod \"must-gather-7nbzj\" (UID: \"9bb854df-b105-409f-9769-081d65756bd0\") " pod="openshift-must-gather-qw6xv/must-gather-7nbzj" Oct 30 14:42:45.230212 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.230188 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9bb854df-b105-409f-9769-081d65756bd0-must-gather-output\") pod \"must-gather-7nbzj\" (UID: \"9bb854df-b105-409f-9769-081d65756bd0\") " pod="openshift-must-gather-qw6xv/must-gather-7nbzj" Oct 30 14:42:45.253972 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.253939 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwwr\" (UniqueName: \"kubernetes.io/projected/9bb854df-b105-409f-9769-081d65756bd0-kube-api-access-5hwwr\") pod \"must-gather-7nbzj\" (UID: \"9bb854df-b105-409f-9769-081d65756bd0\") " pod="openshift-must-gather-qw6xv/must-gather-7nbzj" Oct 30 14:42:45.328508 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.328439 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qw6xv/must-gather-7nbzj" Oct 30 14:42:45.634625 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:42:45.634592 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bb854df_b105_409f_9769_081d65756bd0.slice/crio-348c46fa5f7b44258dfa2fce88678c8db7631aa702be89afc5cf221948da84ce WatchSource:0}: Error finding container 348c46fa5f7b44258dfa2fce88678c8db7631aa702be89afc5cf221948da84ce: Status 404 returned error can't find the container with id 348c46fa5f7b44258dfa2fce88678c8db7631aa702be89afc5cf221948da84ce Oct 30 14:42:45.634625 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.634618 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qw6xv/must-gather-7nbzj"] Oct 30 14:42:45.636818 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:45.636802 4968 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:42:46.268358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:46.268235 4968 scope.go:122] "RemoveContainer" containerID="eebd10a2499d9a1ba086bb0dfa75df0fd57c856708c2e126534829d432d7dce4" Oct 30 14:42:46.279577 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:46.279554 4968 scope.go:122] "RemoveContainer" containerID="2d2027f6653dfb529f3460dc0289cde57b45d46df7a114959466a1eb3daf792a" Oct 30 14:42:46.315187 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:46.315159 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw6xv/must-gather-7nbzj" event={"ID":"9bb854df-b105-409f-9769-081d65756bd0","Type":"ContainerStarted","Data":"348c46fa5f7b44258dfa2fce88678c8db7631aa702be89afc5cf221948da84ce"} Oct 30 14:42:47.324242 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:47.324202 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw6xv/must-gather-7nbzj" event={"ID":"9bb854df-b105-409f-9769-081d65756bd0","Type":"ContainerStarted","Data":"a9651b61e9c54e2f9b994a4e77d3bc9abfde3bcaa8eed7a404572eba31a933db"} Oct 30 14:42:47.324242 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:47.324249 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw6xv/must-gather-7nbzj" event={"ID":"9bb854df-b105-409f-9769-081d65756bd0","Type":"ContainerStarted","Data":"28cf9853bfecbdf78f63a2aff39bd0c8a0fa38e13599290b83326787d11db220"} Oct 30 14:42:47.367633 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:47.367577 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qw6xv/must-gather-7nbzj" podStartSLOduration=2.620597156 podStartE2EDuration="3.367564004s" totalImagesPullingTime="746.966848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:42:47.367204389 +0000 UTC m=+2162.992647761" watchObservedRunningTime="2026-10-30 14:42:47.367564004 +0000 UTC m=+2162.993007378" Oct 30 14:42:49.909844 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:49.909810 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vlmdd_f6ae2497-ab63-4964-940a-e870d9131ae3/global-pull-secret-syncer/0.log" Oct 30 14:42:50.147192 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:50.147152 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dmj28_3829121e-3790-47ff-b827-8774a70fec04/connection-checker/0.log" Oct 30 14:42:50.291668 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:50.291574 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-f298s_a3e4ca01-f835-47a4-b2d4-ff93edaaa6d5/connection-checker/0.log" Oct 30 14:42:50.477440 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:50.477322 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jmrnq_1cdffa3f-7378-4f50-b1bc-e87aa647c4d8/connection-checker/0.log" Oct 30 14:42:50.529047 ip-10-0-129-158 kubenswrapper[4968]: E1030 14:42:50.529008 4968 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmtlt" containerName="extractor" Oct 30 14:42:51.059016 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:51.057610 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h6hnr_ead802d8-7ee2-4381-ad65-723736aabd12/konnectivity-agent/0.log" Oct 30 14:42:51.179908 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:51.173331 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-158.ec2.internal_274ce57e4715bf968542a8e94100b125/haproxy/0.log" Oct 30 14:42:57.172290 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:57.170608 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-cmbdv_64121261-495b-4cfc-855a-edf6f69d62d0/manager/0.log" Oct 30 14:42:57.305461 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:57.305396 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-x8hl6_ad114249-10f3-4069-b4ee-86543a6e1c46/manager/0.log" Oct 30 14:42:57.599771 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:57.598546 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-mlknj_ed0c76b4-f80b-4b08-b3c2-a77524625e1a/manager/0.log" Oct 30 14:42:57.741537 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:57.741496 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-svfmk_e8e6e17d-44a6-4e4e-bba4-236c02d5a856/limitador/0.log" Oct 30 14:42:59.631197 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.631159 4968 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l"] Oct 30 14:42:59.638790 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.638761 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.659474 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.654699 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l"] Oct 30 14:42:59.703736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.703310 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-proc\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.703736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.703367 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-podres\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.703736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.703453 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-lib-modules\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.703736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.703487 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x9s7\" (UniqueName: \"kubernetes.io/projected/93cc5fff-acc4-45f4-be2c-cfb824e06b48-kube-api-access-9x9s7\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.703736 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.703530 4968 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-sys\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.804272 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.804221 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-proc\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.804437 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.804301 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-podres\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.804437 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.804341 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-proc\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.804552 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.804448 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-podres\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.804552 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.804477 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-lib-modules\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.804552 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.804504 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x9s7\" (UniqueName: \"kubernetes.io/projected/93cc5fff-acc4-45f4-be2c-cfb824e06b48-kube-api-access-9x9s7\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.804552 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.804536 4968 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-sys\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.804755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.804554 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-lib-modules\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.804755 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.804622 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93cc5fff-acc4-45f4-be2c-cfb824e06b48-sys\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.828850 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.828820 4968 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x9s7\" (UniqueName: \"kubernetes.io/projected/93cc5fff-acc4-45f4-be2c-cfb824e06b48-kube-api-access-9x9s7\") pod \"perf-node-gather-daemonset-jf94l\" (UID: \"93cc5fff-acc4-45f4-be2c-cfb824e06b48\") " pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:42:59.957456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:42:59.957379 4968 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:43:00.270578 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:00.270552 4968 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l"] Oct 30 14:43:00.272598 ip-10-0-129-158 kubenswrapper[4968]: W1030 14:43:00.272569 4968 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod93cc5fff_acc4_45f4_be2c_cfb824e06b48.slice/crio-064822dcf65c981af79004d38febbbc1c059e388e86f3186941f8cec4095875b WatchSource:0}: Error finding container 064822dcf65c981af79004d38febbbc1c059e388e86f3186941f8cec4095875b: Status 404 returned error can't find the container with id 064822dcf65c981af79004d38febbbc1c059e388e86f3186941f8cec4095875b Oct 30 14:43:00.359203 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:00.358900 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-rt4nm_08b6ecf0-e4e9-46ce-a2ae-6a768ce8be65/cluster-monitoring-operator/0.log" Oct 30 14:43:00.455287 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:00.455243 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" event={"ID":"93cc5fff-acc4-45f4-be2c-cfb824e06b48","Type":"ContainerStarted","Data":"0f5055ab523b24653be812567c41091827932dcb946def8d3de049a85a178bc5"} Oct 30 14:43:00.455287 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:00.455294 4968 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" event={"ID":"93cc5fff-acc4-45f4-be2c-cfb824e06b48","Type":"ContainerStarted","Data":"064822dcf65c981af79004d38febbbc1c059e388e86f3186941f8cec4095875b"} Oct 30 14:43:00.455550 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:00.455372 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:43:00.498104 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:00.498052 4968 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" podStartSLOduration=1.498039655 podStartE2EDuration="1.498039655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:00.497274053 +0000 UTC m=+2176.122717425" watchObservedRunningTime="2026-10-30 14:43:00.498039655 +0000 UTC m=+2176.123483062" Oct 30 14:43:01.885030 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:01.884994 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9ht5_107b3cc3-f2aa-469f-b5a3-6379efd7a53a/node-exporter/0.log" Oct 30 14:43:02.014564 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:02.014529 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9ht5_107b3cc3-f2aa-469f-b5a3-6379efd7a53a/kube-rbac-proxy/0.log" Oct 30 14:43:02.148334 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:02.148261 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9ht5_107b3cc3-f2aa-469f-b5a3-6379efd7a53a/init-textfile/0.log" Oct 30 14:43:06.473649 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:06.473618 4968 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qw6xv/perf-node-gather-daemonset-jf94l" Oct 30 14:43:06.519186 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:06.519159 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-w9s2h_c2536dd8-0f2c-4571-b0db-484894b2a99f/networking-console-plugin/0.log" Oct 30 14:43:07.689858 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:07.689817 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/2.log" Oct 30 14:43:07.694140 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:07.694118 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-84mcb_66b11a57-8746-4a93-b726-a877c2a22a98/console-operator/3.log" Oct 30 14:43:10.238825 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:10.238780 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-nw947_03049b65-310d-4ad7-b614-1daf3a50b528/volume-data-source-validator/0.log" Oct 30 14:43:12.857358 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:12.857313 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fvkvc_17aed83f-3fbe-4b2e-8714-e61122539496/dns/0.log" Oct 30 14:43:12.995370 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:12.995340 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fvkvc_17aed83f-3fbe-4b2e-8714-e61122539496/kube-rbac-proxy/0.log" Oct 30 14:43:13.570958 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:13.570913 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2cgmk_13b59ffb-ff1e-4a12-987b-dc7f5daf9b9b/discovery/0.log" Oct 30 14:43:13.607447 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:13.607423 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-847b5f84b8-7m6sx_9f369d28-d4eb-4d7c-81b6-b80cd88edb80/kube-auth-proxy/0.log" Oct 30 14:43:13.760269 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:13.760236 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qnfg6_78841e01-5bf7-4348-9802-09a81adeb88a/dns-node-resolver/0.log" Oct 30 14:43:14.200456 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:14.200420 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-z2bwp_43dc1f23-75ac-490a-9594-6ba78cb3d01e/istio-proxy/0.log" Oct 30 14:43:14.267984 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:14.267953 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-zl46w_3d721901-63ac-4136-a704-ae8283fce2bb/istio-proxy/0.log" Oct 30 14:43:14.338047 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:14.338021 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-zv6j4_ddf1f9b4-82da-4ad8-a5ee-bebce6df2b3b/istio-proxy/0.log" Oct 30 14:43:14.471536 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:14.471471 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-j8ntq_76bebba1-752e-47cc-b918-89c30bd62cf5/payload-processing/0.log" Oct 30 14:43:14.546854 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:14.546821 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-598ddffcff-nctk8_dfc6f25a-6a7a-472c-af78-683613ffb7a5/router/0.log" Oct 30 14:43:15.439446 ip-10-0-129-158 kubenswrapper[4968]: I1030 14:43:15.439407 4968 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vpkj5_354da4d4-65ce-4f03-a9aa-f7840bf28711/node-ca/0.log"